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ABSTRACT 

Course management soflware enables faculty members to learn one soflware package for web-based curriculum, assessment, 
synchronous and asynchronous discussions, collaborative work, multimedia and interactive resource development. There are as 
many as 109 different course management soflware packages on the market and several studies have evaluated and 
compared various aspects of these tools. However, these studies generally focused on checklists of what these products can do, 
not what these products need to do, or can do well. 

In addition, there is little data to suggest that the pages created by these soflware packages were easy to use by the students, or 
that the use of these tools by faculty increased the quality of instruction experienced by the students. Faculty and students have very 
different requirements for this one product, faculty need to be able to develop resources quickly and efficiently using this soflware, 
while students need to be able to find and access the course materials their instructor has assigned. Therefore, these analyses may 
be of limited predictive power in determining the applicability of different course management soflware for deployment by 
educational institutions. 

The objective of this study was to evaluate the usability and usefulness of course management software to support traditional 
classroom instruction from both the faculty and studenf perspective. This study was done in two parts, the first part asked 
participants with no experience using course management soflware to evaluate several packages and choose the one they 
preferred, and the second part was a follow-up analysis after both faculty and students had used the soflware for an enfire semesfer. 
Faculty members and cadets found that course management soflware [CMS] provided a satisfactory mechanism for web- 
enabled curriculum delivery to supplement traditional classroom instruction at USAFA. If we had only focused on checklists of what 
the soflware products could do, then Blackboard Courselnfo and WebCT would have been considered equal, and Intralearn 
would have been a distant third. However, Blackboard Courselnfo received higher scores than the other products when ratings 
were based on user analysis of ease of use and usefulness. 

Many of the features found in these products such as discussion groups, student web pages and collaborative work areas that 
contributed to their high number of features were not widely used, nor deemed important by both faculty and cadets. Although 
some faculty may use these features in the future (and subsequently cadets will use them] as they become more comfortable and 
familiar with the soflware and pedagogy, at this time a CMS with an easy to use interface that contains a grade book, automated 
quizzes and a place to put announcements and course documents should be preferred to one that contains many collaboration 
features yet also has a difficult navigational interface or hard to use development tools. 

Many faculty members chose to use Blackboard Courselnfo for a follow on semesfer for a more in-depth analysis. Although faculty 
found Courselnfo served most of fheir needs, many indicafed that it lacked some key features necessary for deployment at USAFA, 
for example there was no way to divide grade books, announcements or documents into multiple sections of the same course 
without duplicating the entire course. Similarly, it was not possible to share calendars, quiz questions, handouts, or content among 
instructors teaching the same course. Therefore, this soflware seems besf suifed for providing web-enabled insfrucfional support to 
a faculty member who is teaching one section of a single course, and who does not share large amounts of content with 
instructors teaching other sections. 

As predicted, the faculty and cadefs had different requirements of the CMS product for developing and accessing resources, 
respectively. Therefore, if only the faculty's requirements are considered, or if the product is not evaluated by all of the user groups, 
the probability of purchasing a sub-optimal product for a majority of the users increases. Although the requirements as stated by 
the faculty and cadets who had been using a CMS for an entire semester were more similar than those gathered from the naive 
users, often enterprise CMS purchase decisions are made prior to the purchasing of any system. 

The results obtained in this part of the study will help us determine the requirements for procuring a course management soflware 
system for use here at USAFA. Giving faculty an efficient means of entering their homework assignments into a database system is 
the first and probably most important step to achieving the vision ot an integrated portal system that includes the resources the 
cadets need most, access to homework and reading assignments, course materials and the ability to monitor their progress 
through checking their grades. 
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Section 1. INTRODUCTION 

In the past few years, the World Wide Web (WWW or web) 
and web technologies have become accepted and 
valuable tools for delivering content in distance learning, 
as well as a means to supplement traditional classroom 
instruction (Brown, 2000; Halloran, 1999a; Supinski, 1999). 
The use of the web to support traditional classroom 
instruction ranges from a delivery mechanism for readings 
and course content to a means of expanding the 
curriculum outside the classroom walls, as well as a means 
for engaging students in interactive learning environments 
(Novak et al, 1999). In the past few years, the World Wide 
Web (WWW or web) and web technologies have become 
accepted and valuable tools for delivering content in 
distance learning, as well as a means to supplement 
traditional classroom instruction (Brown, 2000; Halloran, 
1999a; Supinski, 1999). The use of the web to support 
traditional classroom instruction ranges from a delivery 
mechanism for readings and course content to a means 
of expanding the curriculum outside the classroom walls, 
as well as a means for engaging students in interactive 
learning environments (Novak etal, 1999). 

However, one of the drawbacks to web-enabling courses is 
that it may require faculty and students to acquire new 
technical skills, such as the ability to write HTML and other 
web programming language codes or proficiency with 
HTML editor software programs. In these cases, the 
technology can overshadow the teaching and learning of 
the course content (Poe, 2000; Taylor, 2000), and 
converting a course to a web-enabled platform can be 
very labor intensive (Campbell, 2000). 

One possible way to circumvent these drawbacks is to use 
course management system or learning management 
system packages, which use templates specifically 
designed for web-enabled instruction. Course 
management systems CMS) incorporate many different 
development tools into a seamless interface for web 
based course content. By uploading course handouts, 
images and files already created using other software 
packages such as Microsoft (MS) Word or Excel into the 
system, a faculty member can develop a web presence 
without the knowledge of HTML or other languages. 

Course management software enables faculty members 
to learn one software package for web-based curriculum, 
assessment, synchronous and asynchronous discussions. 


collaborative work, multimedia and interactive resource 
development. Tracking features enable instructors to 
examine the use of these online resources, which can be 
used to make inferences about the interest and difficulty of 
the content as well as an individual students' level of effort. 
Although some faculty members may prefer to use 
different software for different applications, incorporating 
all tools into one package may decrease the need to train 
faculty on a variety of software products (Hazari, 2001). 

Deploying a CMS at the institutional level offers students a 
consistent, virtual interactive environment which promotes 
learning and decreases the number of systems a student 
has to learn to accomplish routing tasks, such as finding 
their homework assignments or an instructor's office hours 
(Hazari, 2001). Another advantage is the underlying 
database structure of the system (e.g. Oracle, MySql, etc). 
A homework assignment entered by a faculty member into 
their course can be used to populate a student home 
page through a learning management portal system 
(Halloran, 1999b; and in prep). At the same time, it is 
important that the CMS does not drive the pedagogy and 
limit the creativity of an instructor developing sophisticated 
course content using a variety of different mediums. 

There are as many as 109 different course management 
software packages on the market (London, 2000), and 
several studies have evaluated and compared various 
aspects of these tools (Gray, 1998; Hazari, 2001; InfoWorld, 
1998; Kristapiazzi, 1998; Marshall, 1998). However, these 
studies generally focused on checklists of what these 
products can do, not what these products need to do, or 
can do well. For example, packages were evaluated by 
whether or not they had an asynchronous discussion 
feature, not whether the discussion group feature was easy 
to use or useful to faculty members in teaching their 
courses. Focusing on what a product can do is also of 
limited value given the rapid development and changes in 
product features in this market. 

In addition, there is little data to suggest that the pages 
created by these software packages were easy to use by 
the students, or that the use of these tools by faculty 
increased the quality of instruction experienced by the 
students. Faculty and students have very different 
requirements for this one product, faculty need to be able 
to develop resources quickly and efficiently using this 
software, while students need to be able to find and 
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access the course materials their instructor has assigned. 
Therefore, these analyses may be of limited predictive 
power in determining the applicability of different course 
management software for deployment by educational 
institutions. 

1.1 Objectives of the study 

The objective of this study was to evaluate the usability and 
usefulness of course management software to support 
traditional classroom education from both the faculty and 
student perspective. Faculty were asked to rate the 
different software packages not only on how easy they 
were to use, but how useful the features offered were to 
classroom instruction. Students were asked to evaluate the 
resulting web-enabled courses as to their ease of 
navigation and finding resources as well as how useful they 
found the different features packaged within the products. 
This study was done in two phases, the first phase asked 
participants with no experience using course 
management software to evaluate several packages and 
choose the one they preferred, and the second phase was 
a follow-up analysis after both faculty and students had 
used the software for an entire semester. 

Section 2. COURSE MANAGEMENT SOFTWARE 
USABILITY TESTS 

2.1 Background information 

The United States Air Force Academy (USAFA) is a resident 
institution of higher education. Cadets are required to live 
in dormitories and attend class, and faculty members are 
regularly available to meet with cadets during extended 
office hours. There are 19 academic departments offering 
31 different majors and over 500 different courses. 
Although USAFA does not have a distance-learning 
program, nor offer distance-learning courses, the 
instructors are encouraged to use distance learning and 
other information technologies to supplement the use of 
the traditional classroom education, when appropriate. 
The Academy encourages faculty to make extensive use 
of active-learning strategies for the classroom. 

The Academy first began issuing computers to cadets in 
1985, and since 1989 all cadets have had computers in 
their dormitory rooms. E-mail has been in place at USAFA 
since 1993, and instructors have been able to place class 
handouts on a network of shared computer drives 
accessible by all cadets since 1994. The Academy was 
linked to the internet in 1996, and many faculty now use the 


World Wide Web in their classrooms as a teaching tool. 

With the ubiquitous computing environment here at USAFA, 
there is ample opportunity for faculty members to develop 
resources for their courses. And without coordination 
among departments and instructors on the development 
of web-enabled courses, cadets could end up having to 
learn as many different systems as they take courses. There 
is also a minimum of 24% turnover of the faculty and 
cadets every academic year. Any system deployed here 
must be intuitive in nature, and require minimum training for 
both sets of users, unless provisions are made to 
accommodate extensive training time and costs. 

While it would be possible to author a custom solution CMS, 
the development costs of an enterprise level database 
driven solution are many times more than what the 
commercial off the shelf (COTS) products charge for 
license agreements over many years. In addition, many 
textbook publishing companies are now forming alliances 
with companies that produce CMS software and making 
these supplemental resources available to faculty and 
students who buy their textbooks. In order to take 
advantage of these services, we felt that an exploration of 
COTS CMS solutions was a logical first step. 

Therefore, to accommodate the diversify of courses taught 
here at the Academy, we are interested in a system that 
allows for flexibility as to the type of content that can be 
uploaded by the faculty. However, at the same time, this 
system must be an easy developmental tool for the faculty, 
as well as provide a pedagogically sound and intuitive 
navigational structure for the cadets. Therefore, the 
objective of this portion of the study was to evaluate 
different commercial off the shelf CMS products to begin to 
understand the essential components and requirements 
necessary for deployment of an enterprise level course 
management system at USAFA. 

2.2 Methods 

2.2.1 Selection of software 

For this study, 6 commercial software packages 
were scored using dichotomous (yes/no) 
variables as to whether they have certain 
features or allow for the development of key 
course components. Although some of the 
features evaluated during this process had to do 
with the administration of the software, I focused 
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on those features that were user interface 
components. The two packages with the most 
features were further evaluated for usability. The 
faculty evaluated the product as a 
development tool, whereas the students 
evaluated the resulting courses put together by 
the development tool. 

2.2.2 Participants 

Faculty and cadet members from the United 
States Air Force Academy (USAFA) volunteered to 
participate in the study in response to an email 
query sent to all academic departments. Prior 
to beginning the study they were given a survey 
to assess their computer expertise including any 
prior experience building web pages and using 
the internet both at home and as part of their 
course preparation. They were also asked to rate 
the features that were used in the dichotomous 
rankings of course management software as to 
their usefulness in teaching and learning, prior to 
being exposed to the course management shell 
software. 

2.2.3 Instructor choice experiments 

Instructor study participants were asked to 
develop a web site with a syllabus, course 
information, homework assignments and 
quizzes using two out of three CMS being 
evaluated. Both the software packages and the 
order in which they were presented to the 
instructors was randomized. Instructors created 
course content using the web authoring feature 
of the software package itself, and imported 
pre-manufactured web pages built in plain text 
HTML, MSFrontPage and Macromedia's 
Dreamweaver. They were also given MSWord 
documents and MSExcel spreadsheet files to 
upload to determine the cross compatibility of 
this system with other software products already 
in use at USAFA. The course content, handouts, 
syllabus, quiz questions and quiz answers were 
identical and supplied to the instructor. Therefore 
the ease or difficulty of putting together the 
course web site would be software dependent, 
not dependent on the creativity of the instructor. 

After they had evaluated two CMS packages, 


faculty members were asked to do a side-by- 
side comparison of each product for 18 key 
components. Data collected from the 
instructors included time on task, ease of use, 
and the usefulness of the tasks using a 1-100 
visual analog scale (Miller 1999). They were then 
asked to select one package as the best overall 
and to use that CMS during the Fall '00 semester. 

2.2.4 Cadet choice experiments 

Cadets each evaluated three courses that were 
built by the same person, using the same 
software packages evaluated by the faculty. 
These courses were identical with respect to 
course content, handouts, quiz questions and 
answers, assignments and other information. 
Each cadet evaluated two of the three courses. 
The courses they evaluated as well as the order 
in which they were presented was randomized. 

Cadets were asked to navigate to specific areas 
within the course, take quizzes, participate in 
discussions, collaborate on group assignments, 
find their grade, and perform other tasks 
necessary to test the functional capability of the 
CMS. They were also asked to make predictions 
on where they would find key course 
documents. After they evaluated two courses, 
cadets were asked to complete a side-by-side 
comparison of the courses of each product for 
12 key tasks. Data collected included time on 
task, ease of use and usefulness of the task using 
a 1-100 visual analog scale (Miller, 1999). 
Cadets also selected one course as their 
preferred course, but due to practical 
considerations did not use it for an entire 
semester. 

2.2.5 Data collection and analysis 

Data for both the faculty and cadets were 
collected by self-report on surveys and entered 
into a MS Excel spreadsheet. Summary statistics 
were calculated using MSExcel and are reported 
as mean + sd unless otherwise noted. Data were 
imported into SPSS for further analysis. Non- 
parametric and parametric within subjects 
comparisons were performed to compare 
scores assigned to software packages by faculty 


76 


l-manager’s Journal of Educational Technology, Vol. 2 • No. 1 • April-June 2005 






RESEARCH PAPERS 


Table 1: Dichotomous scoring of web course management software 

Cl = Courselnfo, CT = WebCT, TC = TopCIass, MD = Mad Duck, 
IL = Intralearn, VU = Virtual University 



BB 

CT 

TC 

MD 

IL 

VU 

General Features 







Static toolbar 







Table of Contents 

x 

X 


x 

X 

X 

Customized "look and feel (colors, icons, logos, etc) 

X 


X 

X 


X 

Automated glossary tool 


X 


X 

X 


Automated link to course material content 

X 

X 

X 

X 

X 


Automated indexing tool 

X 

X 




X 

Search tool for course material 

X 

X 

X 


X 

X 

Student can make private annotations of course material 

X 

X 

X 




Integrated Calendar tool 

X 

X 


X 



Multi language support 

X 

X 

X 

X 

X 


Instructor can define groups of students 

X 

X 

X 

X 

X 


- Can assign specific material to individual or group of 

X 

X 

X 

X 



- Collaborative work area for group 

X 

X 



X 


- Group presentation area 

X 

X 





Individual presentation area/homepage 

X 



X 



Student file upload capability/instructor comments 

X 

X 

X 

X 


X 

Conforms to IMS specifications 

X 

X 

X 


X 


Conforms to W3C specifications 





















Authoring features 

BB 

CT 

TC 

MD 

IL 

VU 

Does not require knowledge of FITML 

X 

X 

X 

X 

X 


Allows Java applets 



X 


X 


Requires plug-ins 

no 

no 

no 

no 

no 

no 

Has drag and drop authoring features 

X 






Allows Macromedia files 

X 

X 

X 

X 

X 

X 

Allows Frontpage files 

X 

X 



X 


Allows links to custom developed pages 

X 

X 


X 

X 

X 

Allows custom graphics 

X 



X 

X 
















Course Management Features 

BB 

CT 

TC 

MD 

IL 

VU 

Class list can be entered one student at a time 

X 

X 

X 

X 

X 


Class list can be uploaded as a file 

X 

X 

X 

X 

X 

X 

Class lists can be presented, saved, and printed using 

X 






Online student manual 

X 

X 

X 

X 

X 

X 

Online instructor manual 

X 

X 

X 

X 


X 

Course can be downloaded for safekeeping 





X 


Courses can easily be moved from one server to another 



X 

X 



File upload capability from desktop 

X 

X 

X 

X 

X 

X 

Student tracking features 

X 

X 

X 

X 



- can track how often student accesses pages 

X 

X 


X 



- can track when student accesses pages 

X 

X 


X 



- can track time spent on each page 

X 

X 


X 




and cadets. Unpaired t-tests 
were performed to examine 
differences between computer 
expertise of faculty that finished 
the study with faculty that did 
not finish the study and for 
comparisons between faculty 
and cadets. 

WebCT and Blackboard's 
Courselnfo were housed on a 
Sun Ultra Sparc 5, Intralearn 
software was housed on a 400 
MHz NT 4.0 server with 256 MB 
RAM, and a 4x9 GB SCSI RAID 
(RAID 5) located in the Institute 
for Information Technology 
pplications laboratory in 
Fairchild Hall at USAFA. The web 
pages were accessed hrough 
the academy's intranet 
bypassing the proxies for 
efficiency and speed. 

2.3 Results 

2.3.1 Selection of software for further 
evaluation 

Six software packages were 
dichotomously scored in May 
2000 as to whether they 
contained key pedagogical 
and course management 
system components (Table 1). 

The software packages were 
WebCT, Blackboard Courselnfo 
Level 1, TopCIass, Mad Duck, 

Intralearn, and Virtual University. 

Scoring was done on 
examination copies of the 
software either hosted locally or on the 
company's web servers. These scores were 
compared to other published dichotomous 
scorings of a similar nature (Gray, 1998; Hazarri, 
1999; Kristapiazzi, 1999; Marshall, 1998; Landon, 
2000) for verification. The products with the two 
highest scores (WebCT and Courselnfo) were 


chosen for further evaluations because they 
contained the widest variety of tools for the 
broadest evaluation. Intralearn was also chosen 
because its product uses Microsoft Office 
products for many of its development tools. MS 
Office is the standard suite of tools on the USAFA 
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Communication Features 

BB 

CT 

TC 

MD 

IL 

VU 

One to one course email 

X 

X 

X 

X 

X 

X 

One to many course email 

X 

X 

X 

X 

X 


Searchable asynchronous discussion 

X 

X 



X 


Logged synchronous discussion 

X 

X 



X 


Vitual field trips within discussion pages 

X 






Shared whiteboard 

X 

X 


X 



Adaptable for desktop videoteleconferencing 


3rd 




3rd 















Grading Features 

BB 

CT 

TC 

MD 

IL 

VU 

Student access to progress data available 

X 

X 

X 

X 

X 

X 

Ability to add offline grades 

X 

X 

X 

X 

X 


Grade statistics and/or histograms 


X 

X 




Instructor comments available with grade 

X 

X 

X 




Scores can be emailed to instructor 




X 



Scores can be stored on server 

X 

X 


X 



Scores can be exported into Excel 

X 

X 


X 

















Quizzing features 

BB 

CT 

TC 

MD 

IL 

VU 

Quizzes automatically graded and entered into 

X 

X 


X 

X 

3rd 

Allows for the following automatically graded question 






3rd 

- True -False/Muitiple choice 

X 

X 

X 

X 

X 

3rd 

- Fill in the blank 

X 

X 


X 

X 

3rd 

- List matching 

X 

X 




3rd 

- Essay questions 

X 

X 




3rd 

- Imagemap (click on correct part of image) 







- Short answer 

X 

X 





Questions can have multiple correct answers 

X 

X 

X 

X 



Can use a mixture of question types on a single quiz 

X 

X 

X 


X 


One question at-a-time testing capability 



X 


X 


Question file upload capability 

X 

X 

X 


X 


Customized feedback 


X 

X 

X 



Redirect path dependent on question answers 


X 

X 

X 

X 


Timed quizzes 

X 

X 

X 

X 

X 


Delivered on-line on a predetermined time and day 


X 





Supports graphics files adjacent to quiz question 

X 

X 

X 




Supports both tutorial and real exam scenarios 

X 

X 





Random assignment of questions to exams 

X 

X 

X 

X 

X 


Allows weighting of questions so students get equal 







Has database application for quiz question bank 

X 

X 

X 











Can be used with the following server /Operating 

BB 

CT 

TC 

MD 

IL 

VU 

Unix 

X 

X 

X 

X 


X 

NT 

X 

X 

X 

X 

X 

X 

Macintosh 



X 

X 



Solaris 

X 

X 

X 


X 

X 

Linux 

X 

X 

X 



X 

Other 














Total 

60 

59 

42 

43 

36 

17 


computer desktop and therefore would provide a control for 
familiarity. 

2.3.2 Participants 

Twenty-eight faculty members from the United States Air Force 
Academy volunteered to evaluate the course management shell 
software during July and August of 2000. These faculty were a 


representative sample of all USAFA 
faculty and consisted of members 
from all four academic divisions 
and staff (Table 2). There were six 
females and 22 males whose 
average age was 38.4 + 8.9. 

Table 2: Demographics of the faculty participants 
compared to the demographic makeup 
of all USAFA faculty. 



Participants 

USAFA Facuity 


N 

% 

N 

% 

Mde 

22 

78.6 

429 

81.0 

Femde 

6 

21.4 

102 

19.0 






Military 

17 

60.7 

398 

75.0 

Civilian 

11 

39.3 

133 

25.0 






Academic Divisions: 





Humanities 

6 

21.4 

104 

19.6 

Social Sciences 

6 

21.4 

172 

32.3 

Basic Sciences 

7 

25.0 

135 

25.5 

Engineering 

6 

21.4 

120 

22.6 

Other 

3 

10.8 








Total 

28 

100.0 

531 

100.0 


Fifteen of these participants 
had Master's degrees 
(MA/MS/MPH) and 12 had a 
Ph. D or other terminal degree. 

Thirty cadets from USAFA volunteered to 
evaluate courses produced by the 
course management shell software 
during October 2000. These cadets were 
a representative sample of all cadets at 


Table 2: Demographics of the faculty participants 
compared to the demographic makeup 
of all USAFA faculty. 



Participants 

Ail Cadets 


N 

% 

N 

% 

Male 

26 

89.7 

3557 

84.7 

Female 

3 

10.3 

643 

15.3 






Year at the Academy 





First Class 

4 

13.8 

874 

20.8 

Second Class 

7 

24.1 

962 

22.9 

Third Class 

10 

34.5 

1147 

27.3 

Fourth Class 

9 

31.0 

1218 

29.0 






Academic DMsIon: 





Humanities 

0 

0.0 

239 

5.7 

Social Sciences 

10 

34.5 

1189 

28.3 

Basic Sciences 

8 

27.6 

521 

12.4 

Engineering 

6 

20.7 

802 

19.1 

Undecided 

6 

20.7 

1142 

27.2 

Interdisciplinary 



307 

7.3 






Total 

29 

100.0 

4200 

100.0 
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2.3.3 Computer experience 

Faculty participants were experienced 
computer users familiar with web browsing. The 
average participant had 12.7 + 4.8 years 
experience using computers, with 5.6 + 7.3 
years experience programming computers. 
They averaged 5.1 + 1.8 years of web browsing 
experience and routinely browse the web 5.1 + 
4.4 hours a week for both professional and 
personal use (Table 4). 

Faculty participants were familiar with web- 
based curriculum materials. Seventy-eight 
percent of the initial participants reported 
having used web-based curriculum support 
materials at some time in the past, with 59% of 
the participants using web-based materials for 
the Fall '00 semester. Of those who used web- 
based materials, over half reported using 


resources they developed (31% of all 
participants). Twenty-two percent have 
maintained a personal web server and 30% of 
all participants maintain a personal website with 
an internet service provider outside USAFA. 

Of those who have developed their own web- 
based content, they reported 1.5 + 1.7 years 
experience developing those materials. 
However, the faculty had little experience using 
HTML authoring tools (0.6 + 0.9 years 
experience) such as MS Frontpage, Netscape 
Composer and Macromedia Dreamweaver[l]. 
Faculty members reported no experience using 
interactive development tools such as Allaire's 
Cold Fusion, Microsoft Visual Studio, 
Macromedia Director or Macromedia Attain 
objects. The faculty also had no experience 
using any CMS products such as those being 
tested in this study (Table 4). 


Table 4: A comparison of USAFA faculty (n = 28) and cadet (n = 30) computer experience. 


Task 

Faculty 

Cadets 

t 

df 

P 

Yecrs using computers 

12.7 

9.7 

2.73 

56 

0.008 

Yeas web browsing 

5.1 

5.5 

0.89 

56 

0.375 

Hons per week brewsing the web 

5.1 

11.7 

3.66 

43.1 3 

Q001 

Yeas developing web oontent 

1.5 

1.5 

Q01 

56 

0.993 

Yeas computer progamring 

5.6 

2.0 

2.50 

31.5 s 

0.018 

Yeas teaching with vwwv-bcsed curicUim rraferids 1 

5.1 





Yeas developing www-based curicUim rmderids 1 

1.8 





Number of ocuses using vwwwbcBed curicUim rraterids 1 


3.0 




Mdntan a persond web sever at USAFA (percent) 

18.0 

10.0 

0.86 

56 

0.395 

Mdntan a pesend website outside USAFA (percent] 

30.0 

40,0 

Q81 

55 

0.422 

Yeas pregerrmng HTTVL 

0,3 

0.9 

1.63 

363? 

am 

Yeas pregerrmng JAVA 

0.1 

0.3 

1.13 

31.6 3 

0.269 

Yeas progcrrrri ng javcBcript 

0.0 

0.3 

1.80 

31HJ 

0081 

Yeas using HIM. development saftwere 2 

0.4 

0.5 



n.s 

Yeas using web development softwae 2 

0.0 

0.1 



n.s 

Web-based Gdlcbaation Tods 2 

0.1 

0.0 



n.s 

Oouse Margement Tods 2 

0.0 

0.0 



n.s 


statistical tests were not performed due to a lack of comparable question 

statistical tests were not performed due to inadequate sample size 

the degrees of freedom were adjusted for unequal variance among groups 


Cadet participants were very familiar with using 
web-based curriculum materials. They reported 
that3.0 + 1.3 oftheir instructors used web-based 
curriculum support materials for their courses in 
Fall '00. Cadets have 1.5 + 2.0 years experience 
developing their own web-based content. 


Twenty percent have at one time maintained a 
personal web server and 40% of all the cadets 
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Macromedia Director or Macromedia Attain 
objects. The cadets had no experience with web- 
based curriculum resources built by CMS products 
such as those being tested for this study. 

When faculty members' computing experience 
was compared to cadet computing 
experience, faculty members had significantly 
more experience using and programming 
computers than did cadets. However, cadets 
spend significantly more time browsing the 
internet each week than do faculty. There was no 
significant difference between cadets and 
faculty in the number of years each group 
reported browsing the web, or developing www 
content. 

2.3.4 Faculty who finished compared to those who did 
not finish the experiments. 

Of the 28 faculty members who started the 
experiments, 15 faculty finished the experiments 
and 13 of the faculty did not finish the 
experiments. Faculty who finished the 
experiments did not have significantly more 
computer or web development experience 
than those faculty who did not finish the 
experiments for 11 different experience 
parameters evaluated (Table 5). Faculty who did 
not finish the experiments cited personal time 
management issues as the main reason for 
discontinuing the experiments. 

2.3.5 Rankings of selection criteria by faculty and 
cadets 

When faculty and cadets were asked to rank 
the features used to evaluate the course 
management software packages (Table 1), 


they gave most of the features similar scores. 
The Grand mean rating given to all the features 
by faculty was 4.6 + 0.70 and for cadets was 
4.1 + 0.82 on a 6 point scale, where 1 
indicated that the feature was of very little 
importance, and 6 was very important or a 
"must have". Cadets and faculty disagreed on 
which features were more and less important, 
with the exception of one feature in each 
category (Table 6 and 7). 

Nine features were rated 5.3 or higher and 
considered to be more important to faculty 
developing curriculum. Most of these features 
were course administration features or the 
ability to include HTML content developed 
independently of CMS software (Table 6). Five 
features received a score of 3.9 or below and 
were considered less important including the 
ability to conform to external web 
development specifications such as IMS, the 
use of discussion groups, video¬ 
teleconferencing and virtual field trips. The 
ability to support foreign languages and the 
type of operating system used to run the 
software received scores less than 3.2 and 
were designated as least important (Table 6). 
However, the ability to support foreign 
languages was rated higher by the foreign 
language faculty, receiving a mean rating of 
5.75onascale of 6. 

Eight features were rated 5.0 or higher and considered to 
be more important than the others to cadets for web- 
enabled curriculum. The features included in this group 
were the ability to access grades and navigational 
features such as a search tool, online student manual, and 


Variable 

Finished 

DNF 

t 

df 

P 

Years using computers 

12.4 

13.0 

0.32 

26 

0.750 

Years web browsing 

5.5 

4.7 

1.24 

26 

0.224 

Years developing web content 

1.4 

1.5 

0.15 

26 

0.886 

Years using ftp 

2.1 

1.9 

0.19 

26 

0.852 

Years computer programming 

5.9 

5.2 

0.23 

26 

0.823 

Years developing www-based curriculum 

0.9 

0.7 

0.68 

26 

0.299 

Maintain a personal web server at USAFA 

13.0 

23.0 

0.65 

26 

0.520 

Maintain a pesonal web site outside USAFA (%) 

33.0 

23.0 

0.58 

26 

0.566 

Years programming HTML 

0.4 

0.5 

0.32 

26 

0.755 

Years programming JAVA 

0.0 

0.1 

0.55 

26 

0.590 

Years programming javascript 

0.1 

0.3 

0.82 

26 

0.421 


Tabl e5: Comparison of 
computer 
experience for 
faculty who finished 
the evaluation of the 
learning 
management 
software [n=15], with 
faculty who did not 
finish [DNF] the 
evaluation (n=13 ]. 
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Table 6: The mean ratings of CMS features faculty (n = 29) rated significantly higher or lower than the 
grand mean for all features. The corresponding mean scores given by cadets (n = 30) are also 
shown. Features that were given equivalent ratings by both groups are shown in bold. 
The ratings were on a 6 point scale where 1 indicated that the feature was of very little 
importance to web-enabled curriculum, and 6 was very important or a "must have". The 
grand mean score for all features was 4.6 + 0.7 for faculty and 4.1+0.82 for cadets. 


Feature 

Faculty 

s.d. 

Cadet 

s.d, 

more important: 





Course can be downloaded for safekeeping 

5.4 

0.8 

4.9 

1.2 

Can use a mixture of question types on a single quiz 

5.4 

0.7 

4.7 

1.1 

Automated link to course material 

5.3 

0.9 

5.2 

1.1 

Allows links to custom developed pgs 

5.3 

0.7 

4.6 

1.5 

Allows custom graphics 

5.3 

0.8 

4.4 

1.5 

Class list entered one at at time or uploaded as file 

5.3 

1.1 

4.6 

1.3 

Class lists can be presented, saved, printed using Excel 

5.3 

1.1 

4.7 

1.4 

Quizzes automatically, graded and entered into 

5.3 

1 

4.9 

1.3 

Question file upload capability 

5.3 

0.7 

4.1 

1.6 

less important: 





Logged synchronous discussion 

3.8 

1.3 

3.3 

1.7 

Virtual field trips within discussion pages 

3.8 

1.4 

3.8 

1.4 

Adaptable tor desktop videoteleconferencing 

3.8 

1.3 

3.7 

1.8 

Requires plug-ins 

3.5 

1.7 

3.4 

1 

Conforms to IMS specifications 

3.4 

1.9 

3 1 

1.5 

least important: 





Support foreign languages 

3 

1.9 

3 

1.6 

Operating System 

2.7 

2 

2.4 

1.7 


static toolbar (Table 7). Six features 
received a score of 3.2 or below 
and were considered less 
important, including the ability to 
conform to external web 
development specifications such 
as IMS, the ability to support 
foreign languages, and the ability 
to track students access to the 
materials. The operating system 
used to run the software received 
a score of 2.4 and was 
determined to be least important 
to the cadets (Table 7). 

2.3.6 Faculty evaluation of 
the software for course 
development 

Ten faculty members 
finished evaluations 
comparing Courselnfo 
to WebCT, and 5 faculty 
finished evaluations 


Table 7: The mean ratings of course management software features that cadets (n = 30) rated 
significantly higher or lower than the grand mean for all features. The corresponding mean 
scores given by faculty (n=29) are also shown. Features that were given equivalent ratings by 
both groups are shown in bold. The ratings were on a 6 point scale where 1 indicated that the 
feature was of very little importance to web-enabled curriculum, and 6 was very important or 
a "must have". The grand mean score for all features was 4.6 + 0.70 for faculty and 4.1+0.82 
for cadets. 


Feature 

Cadet 

s.d. 

Faculty 

S.d. 

more important: 





Student access to progress data available 

5.6 

0.7 

5.0 

1.1 

Online student manual 

5.4 

0.7 

4.8 

1.2 

Search tool for course material 

5.4 

0.9 

5.1 

0.7 

Table of contents 

5.3 

0.8 

5.1 

1.3 

Automated link to course material content 

5.2 

1.1 

5.3 

0.9 

Instructor comments with grade 

5.1 

1.2 

5.0 

1.1 

Static toolbar 

5.1 

1.1 

4.0 

1.8 

True-False/Multiple choice quiz questions 

5.1 

1.3 

5.2 

0.9 

List matching quiz questions 

5.0 

1.3 

5.2 

0.8 

less important: 





Conforms to IMS specifications 

3.1 

1.5 

3.4 

1.9 

Can support foreign languages 

3.0 

1.6 

3.0 

1.9 

Can track how often student accesses pages 

3.0 

1.5 

4.7 

1.2 

Can track time spent on each page 

3.0 

1.5 

4.4 

1.1 

Student tracking features 

3.0 

1.5 

5.0 

1.3 

Can track when student accesses pages 

2.9 

1.4 

4.6 

1.1 

Least immportant: 





Operating system (except for NT and linux 

2.4 

1.7 

2.7 

2.0 
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comparing Courselnfo to Intralearn. All 15 
participants in both groups preferred Courselnfo 
over the other products evaluated, irrespective 
of which package they tried first. Given the 
unanimous choice of Courselnfo and time 
constraints of the faculty, we did not run a WebCT 
vs Intralearn comparison group and stopped 
recruiting more faculty members to compare 
Courselnfo with Intralearn once the minimum 
sample size of 5 individuals had been reached 
(Nielsen, 2000). In addition, since all faculty 
chose the same product, analyses to determine 
if preference, ease of use and time on task was 
dependent upon presentation order was not 
done. 

Of the 15 faculty members who finished the 
experiments, three chose provide written 
commentary instead of completing the 
comparison surveys. Therefore, their data are not 
represented here for time on task, ease of use, or 
usefulness of the features evaluated. There was 
no significant difference between the scores 
given to Intralearn and WebCT for features they 
shared in common, so those data have been 
pooled to simplify the analyses. 

2.3.6a Time on task 

It took faculty 1 hour and 31 minutes on average 
to create a web-enabled course using 
Courselnfo and 2 hours and 22 minutes on 
average to create a course identical in content 
using either WebCT or Intralearn (Table 8). 
However, this difference was not statistically 
significant due to inter-individual differences 
among participants. There was no significant 
difference in the amount of time it took to 
upload assignments, make quizzes, do simple 
administrative tasks or customize the look and 
feel of the courses between Courselnfo, WebCT 


and Intralearn , however it took them only one- 
third the time to set up a course using Courselnfo 
than the other products. 

2.3.6b Ease of Use 

Courselnfo was rated significantly easier to use 
by faculty than the other software packages in 

Table 8: The mean amount of time it took faculty at USAFA (n=12] 
to create a web-enabled course using one of three CMS 
software packages. Cl = Courselnfo and Other = Intralearn 
and WebCT. 


Task 

Cl 

Other 

t 

df 

P 

Setup 

0:18 

0:55 

2.20 

7 

0.063 

Assignments 

0:20 

0:31 

1.61 

8 

0.145 

Quizzes 

0:12 

0:15 

0.81 

9 

0.438 

Administration 

0:37 

0:40 

0.23 

6 

0.823 

Miscellaneous 

0:24 

0:30 

0.45 

6 

0.666 

Customization 

0:04 

0:11 

1.43 

3 

0.248 

Total 

1:31 

2:22 

1.54 

9 

0.158 


side-by-side comparisons for 16 of the 18 tasks 
evaluated including creating announcements, 
composing quizzes, adding assignments, using 
collaboration features and doing administrative 
tasks (Table 9). All of the features were rated as 
somewhat easy to very easy with the exception 
of enrolling the cadets in class, exporting the 
gradebook into MS Excel, file sharing or using the 
group workspace, which were rated as neutral. 
The faculty scored nine features as difficult to 
somewhat difficult and none of the features as 
somewhat easy to very easy for WebCT and 
Intralearn (Table 9). However, faculty members 
did not use the online manual or ask for help 
significantly more often when using WebCT or 
Intralearn than they did when using Courselnfo 
(Table 10). 

The faculty also rated Courselnfo as being the 
most consistent with how other HTML documents 
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Table 9: Comparison of faculty (n = 12) mean ease of use scores for Courselnfo (Cl) and ofher CMS 
producfs (WebCT and Infralearn). Anchors for fhis scale are 0 = very difficult 12 = difficult 
29 = somewhat difficult 66 = somewhat easy 82 = easy 100 = very easy 


Task 

Cl 

Other 

t 

df 

P 

Course set up 

86.9 

28.3 

8.49 

11 

< 0.001 

Adding assignments 

85.1 

37.3 

6.06 

11 

< 0.001 

Uploading web pages 

79.2 

34.0 

4.22 

11 

< 

Linking other web pages 

74.1 

24.4 

4.18 

11 

< 

Composing quizzes 

84.1 

43.5 

5.02 

11 

< 0.001 

Aministrative functions overall 

77.6 

31.9 

9.76 

11 

< 0.001 

Enrolling cadets 

56.1 

23.7 

4.96 

11 

< 0.001 

Creating announcements 

81.2 

46.2 

4.13 

11 

< 

Sending email to class 

83.3 

28.0 

7.19 

11 

< 0.001 

Setting up grade book 

71.5 

15.4 

6.67 

11 

< 0.001 

Entering grades into grade book 

76.1 

22.9 

5.34 

11 

0.017 

Exporting gradebook into excel 

62.0 

18.0 

3.10 

11 

0.011 

Using discussion groups 

72.1 

31.4 

3.20 

11 

< 0.001 

File sharing 

46.9 

26.8 

1.44 

11 

0.193 

Group workspace 

40.9 

11.6 

2.82 

11 

< 0.001 

Calendar 1 

77.9 

34.4 

2.89 

8 

0.020 

Chat rooms 

71.6 

31.3 

4.01 

11 

0.003 

Modifying look and feel of course 1 

68.1 

45.3 

3.17 

8 

0.110 


Intralean did not have this feature, so this value represents a comparison between Courselnfo and WebCT only. 


Table 10: Number of times faculty (n = 12) participants accessed a 
manual or asked for help when using a one of fhree course 
management software products. Cl = Courselnfo and 
Other = Intralearn and WebCT. 


Type of help 

Cl 

Other 

t 

df 

P 

Use online manual 

3.2 

3.4 

0.17 

9 

0.872 

Asked questions 

0.2 

0.7 

1.25 

9 

0.244 


were displayed when uploaded into the system. 
All of the packages caused one of the animated 
pages to have misaligned graphics, but 66% of 
the pages appeared the same before and after 
they were uploaded into Courselnfo, whereas 
only 50% of the pages appeared the same 
when uploaded into WebCT. None of the 
participants could find any graphics on the 
pages when they were uploaded into Intralearn.] 
2.3.6c Usefulness 

The faculty rated the tasks they were asked to 


perform using Courselnfo significantly more 
useful to teaching than the tasks they were asked 
to complete with the other software packages in 
side-by-side comparisons for 17 of the 18 tasks 
evaluated, even though the tasks they were 
asked to do with each product were identical. 
These tasks included creating announcements, 
setting up the grade book, and uploading web 
pages created using other software products. 
(Table 11). All of the features were rated as 
somewhat useful to very useful by the faculty 
using Courselnfo with the exception of discussion 
groups, the group workspace, chat rooms and 
modifying the look and feel of the course, which 
were rated as neutral. The faculty scored none of 
the 18 tasks as somewhat useful to very useful for 
WebCT and Intralearn, but scored 16 of the tasks 
as neutral, and chat room and discussion 
groups of limited use (Table 11). 

Since the tasks they were asked to perform with 
each CMS package were the same, faculty 
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Table 11: Comparison of mean usefulness scores given by faculty (n = 12) for Courselnfo (Cl) and 
other CMS products (WebCT and Intralearn). Anchors for this scale are 0 = not at all 
useful, 12 = not very useful, 29 = of limited use, 66 = somewhat useful, 82 = useful, 
100 = very useful. 


Task 

Cl 

Other 

t 

df 

P 

Course set up 

84.1 

50.1 

2.82 

11 

0.230 

Adding assignments 

87.5 

44.1 

3.18 

11 

0.015 

Uploading web pages 

92.1 

35.7 

3.48 

11 

0.008 

Unking other web pages 

74.0 

36.2 

2.97 

11 

0.018 

Composing quizzes 

88.1 

49.0 

3.05 

11 

0.019 

Administrative functions overall 

80.4 

54.0 

2.78 

11 

0.024 

Enrolling cadets 

76.8 

43.4 

2.71 

11 

0.027 

Creating announcements 

83.6 

51.6 

2.89 

11 

0.018 

Sending email to class 

80.1 

38.4 

3.32 

11 

0.011 

Setting up grade book 

79.6 

44.6 

2.80 

11 

0.021 

Entering grades into grade book 

78.6 

51.6 

2.31 

11 

0.046 

Exporting gradebook into excel 

79.5 

48.3 

2.37 

11 

0.050 

Using discussion groups 

53.7 

26.1 

2.44 

11 

0.040 

File sharing 

74.4 

39.0 

3.02 

11 

0.017 

Group workspace 

63.3 

33.4 

3.01 

11 

0.017 

Calendar 1 

73.3 

36.8 

2.80 

8 

0.020 

Chat rooms 

53.8 

25.0 

3.01 

11 

0.017 

Modifying look and feel of course 1 

62.8 

29.9 

3.17 

8 

0.011 


Intralean did not have this feature, so this value represents a comparison between Courselnfo and WebCT only. 


Courselnfo was rated 
significantly higher than the 
other software packages for 10 
statements designed to 
ascertain their overall 
experience using the software 
product. Faculty felt that 
Courselnfo had a significantly 
better navigational layout was 
more intuitive in nature and 
required little training to use, as 
well as had pedagogically 
useful features. All statements as 
they applied to Courselnfo were 
rated as moderately agree to 
decidedly agree, except for the 


should have given them the same scores, 
therefore it appears the scores that the faculty 
gave the tasks were more dependent upon the 
software package or whether they were easy to 
do, than how useful these tasks were to teaching. 
A regression analysis of ease of use and 
usefulness scores assigned for both software 
packages indicates that scores that faculty 
assigned for usefulness were significantly 
dependent upon the scores faculty assigned for 
ease of use (r = 0.8, p < 0.001, df = 35). 


online user's manual and the 
directions given on the screen. 
The faculty scored seven of the 
10 statements features as 
moderately disagree to 
decidedly disagree for WebCT 
and Intralearn including the 
navigational layout, the 
pedagogical layout, was easy 
to use, and needs little training 
to use (Table 12). 


2.3.6d Overall evaluation by faculty 


2.3.7 Evaluation of CMS courses by 
cadets 


Table 12: Comparison of mean overall assessment scores by faculty (n = 12) for Courselnfo (Cl) 
and other CMS products (WebCT and Intralearn). Anchors for this scale are 0 = 


Twenty nine of the thirty cadets 
finished the evaluation of the 


decidedly disagree, 25 = moderately disagree, 50 = neutral, 75 = moderately agree, 
100 = decidedly agree. 


Statement 

Cl 

Other 

t 

df 

P 

Took a reasonable amount of time 

76.4 

40.0 

2.45 

11 

0.034 

Had a good navigational layout 

80.0 

24.1 

4.43 

11 

0.001 

Had a good pedagogical layout 

73.5 

24.0 

4.92 

11 

0.001 

Had aesthetically pleasing screen displays 

75.0 

48.6 

2.97 

11 

0.015 

Was easy to use 

83.6 

9.6 

11.20 

11 

0.001 

Had an easy to use manual 

66.0 

18.5 

5.16 

11 

0.001 

Had instructive directions on the screen 

67.7 

23.2 

5.15 

11 

0.001 

Had pedagogically useful features 

80.0 

40.0 

4.15 

11 

0.002 

Was intuitive in nature 

83.6 

12.7 

9.99 

11 

0.001 

Needs little training to use 

79.6 

9.1 

9.05 

11 

0.001 


course management software 
products. Of the ten cadets who 
evaluated courses built with 
Courselnfo and WebCT, seven 
preferred courses built with 
Courselnfo and only three 
preferred courses built with 
WebCT. Of the nine cadets who 
evaluated courses built with 
Courselnfo and Intralearn, all 
nine preferred the courses built 
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with Courselnfo. And of the ten cadets who 
evaluated courses built with Intralearn and 
WebCT, five preferred courses built with Intralearn 
and five preferred courses built with WebCT. 

There was no preference for software package 
based on the order in which the products were 
evaluated. Cadets chose the first package that 
they were presented 44.8% and the second 
package they were presented 55.2% of the 
time. Since the courses were presented to the 
cadets according to a randomization scheme, 
each course had an equal chance of being 
presented first or second. 

To evaluate the courses, cadets located 
announcements and the syllabus, submitted 
assignments, accessed external links, used the 
discussion boards, took quizzes and checked 
their grades. Cadets were given the answers to 
the quizzes, and the quizzes, syllabus and 
supporting materials were identical for each 
course, however each course had a different 
navigational interface. The interface for 
Courselnfo and WebCT required the cadets to 
search for assignments for a particular lesson 
under different course headings, whereas the 
navigational layout for Intralearn had the daily 
lessons along with the assignments listed on the 
opening page. 

2.3.7a Time on task 

Cadets spent an average of 26.6 minutes 
evaluating each course. There was no significant 
difference for the overall amount of time that it 
took them to complete the tasks for each course 
(Table 13 a-c), but it took the cadets significantly 
less time to complete quizzes in the course built 
with Courselnfo than WebCT, and significantly 
less time to find the syllabus and supporting 
documents in the course built with Courselnfo 
than with Intralearn. Cadets spent statistically 
significant less time completing the evaluation 
on the second software package than they did 
the first software package (Table 14). 

2.3.7b Ease of Use 


Table 13a: The mean amount of time in minutes it took cadets at 


USAFA [n=10] to navigate and access materials in web- 
enabled courses created with CI(Courselnfo] and 
WebCT. 


Task 

WebCT 

Cl 

t 

df 

P 

Locating course documents 

6.5 

6.5 

0.00 

8 

1.00 

Completing assignments 

8.5 

7.4 

0.40 

8 

0.60 

Accessing external links 

1.1 

0.9 

1.31 

8 

0.22 

Using communication tools 

6.2 

6.2 

0.00 

8 

1.00 

Taking quizzes 

7.0 

5.0 

3.32 

8 

0.01 

Total time 

26.5 

23.5 

0.77 

9 

0.45 


Table 13b: The mean amount of time in minutes it took cadets at 
USAFA (n=9] to navigate and access materials in web- 
enabled courses created with CI(Courselnfo] and 
Intralearn. 


Task 

Cl 

Intralearn 

t 

df 

D 

Locating course documents 

4.8 

7.8 

3.00 

8 

0.01 

Completing assignments 

5.5 

6.9 

0.70 

7 

0.49 

Accessing external links 

0.9 

1.8 

1.45 

8 

0.18 

Using communication tools 

5.3 

5.1 

0.21 

8 

0.83 

Taking quizzes 

5.1 

5.4 

0.10 

7 

0.91 

Total time 

20.6 

26.9 

1.37 

8 

0.20 


Table 13c: The mean amount of time in minutes it took cadets at 
USAFA (n=10] to navigate and access materials in web- 
enabled courses created with WebCT and Intralearn. 


Task 

WebCT 

Intralearn 

t 

df 

D 

Locating course documents 

6.5 

9.2 

1.71 

9 

0.11 

Completing assignments 

11.0 

8.6 

0.64 

9 

0.53 

Accessing external links 

1.5 

2.2 

1.73 

9 

0.11 

Using communication tools 

4.7 

7.9 

1.12 

9 

0.29 

Taking quizzes 

5.6 

4.3 

1.74 

8 

0.11 

Total time 

30.4 

31.8 

0.17 

9 

0.86 


Table 14: The mean amount of time in minutes it took cadets at 
USAFA (n=29] to navigate and access materials in web- 
enabled courses created with CMS software by order in 
which the course was presented to the students. 


Task 

first 

second 

t 

df 

P 

Locating course documents 

8.0 

5.9 

2.74 

27 

0.01 

Completing assignments 

9.4 

6.9 

1.63 

26 

0.11 

Accessing external links 

1.5 

1.3 

0.90 

27 

0.37 

Using communication tools 

6.6 

5.3 

1.06 

27 

0.29 

Taking quizzes 

6.3 

4.8 

1.20 

26 

0.23 

Total time 

30.2 

22.3 

2.25 

28 

0.03 


There was no significant difference between 
courses built with Intralearn and WebCT, or 
between courses built with Courselnfo and 
WebCT for ease of use scores that cadets 
assigned to 12 different tasks (Tables 15a and c). 
Cadets rated courses built with Courselnfo 
significantly easier than Intralearn for 5 of the 12 
tasks evaluated (Table 15b). The cadets rated 
most of the features as somewhat easy to very 
easy to use. The only features that were found 
difficult for all of the courses was using the drop 
box to submit assignments to the instructor. 
Cadets also rated using the discussion boards in 
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the WebCT course somewhat difficult as well as 
locating announcements, checking their 
grades and using the calendaring function in the 
Intralearn course, (note: Intralearn does not have 
a calendarfunction). 


Table 15a: Comparison of cadet (n = 10] mean ease of use scores 
for courses built with Intralearn and WebCT. Anchors for 
this scale are 0 = very difficult 12 = difficult 
29 - somewhat difficult 66 = somewhat easy 82 = easy 
100 = very easy 


Task 

Intralear 

WebCT 

t 

df 

P 

Locating the syllabus 

748 

822 

066 

9 

0.524 

Locating the announcements 

36.9 

66.7 

2.13 

6 

0.077 

Locating instructor information 

89.2 

89.4 

0.08 

9 

0.939 

Locating the course policies 

90.1 

89.9 

0.08 

9 

0.939 

Locating the assignments 

79.6 

78.9 

0.16 

9 

0.876 

Using the drop box 

26.0 

37.6 

083 

6 

0.437 

Locating external links 

68.2 

89.9 

1.93 

9 

0.086 

Using the discussion boards 

64.4 

63.6 

0.10 

9 

0.922 

Using the calendarfunction 

27.6 

87.4 

3.10 

4 

0.036 

Taking quizzes 

60.5 

78.0 

207 

7 

0.077 

Checking your grade 

53.6 

73.4 

141 

7 

0.201 

Emailing the instructor 

80.9 

76.6 

0.28 

8 

0.784 


Ta b I e 15 b: Comparison of cadet (n = 9] mean ease of use scores for 
courses built with Courselnfo [Cl] and Intralearn. Anchors 
for this scale ore 0 = very difficult 12 = difficult 
29 = somewhat difficult 66 = somewhat easy 82 = easy 
100 = very easy. 


Task 

Cl 

Intralear 

t 

df 

P 

Locating the syllabus 

89.0 

85.6 

0.96 

8 

0.364 

Locating the announcements 

94.4 

55.7 

2.49 

6 

0.047 

Locating instructor information 

90.7 

80.3 

2.13 

8 

0.066 

Locating the course policies 

91.0 

81.0 

1.39 

8 

0.202 

Locating the assignments 

94.8 

86.7 

1.44 

8 

0.188 

Using the drop box 

69.5 

32.3 

2.38 

7 

0.049 

Locating external links 

95.8 

70.9 

2.73 

8 

0.026 

Using the discussion boards 

95.6 

77.4 

2.55 

8 

0.034 

Using the calendarfunction 

90.0 

5.3 

13.83 

3 

0.001 

Taking quizzes 

93.8 

80.8 

1.51 

8 

0.170 

Checking your grade 

82.5 

55.4 

1.18 

7 

0.277 

Emailing the instructor 

94.4 

66.6 

2.12 

8 

0.067 


Table 15c: Comparison of cadet (n = 10] mean ease of use scores 
for courses built with Courselnfo [Cl] and WebCT. Anchors 
for this scale ore 0 = very difficult 12 = difficult 
29 = somewhat difficult 66 = somewhat easy 82 = easy 


100 = very easy. 


Task 

Cl 

WebCT 

t 

df 

P 

Locating the syllabus 

78.7 

82.4 

0.56 

9 

0.588 

Locating the announcements 

94.5 

92.8 

0.41 

9 

0.692 

Locating instructor information 

93.5 

93.7 

1.00 

9 

0.343 

Locating the course policies 

92.7 

94.2 

0.71 

9 

0.496 

Locating the assignments 

94.7 

90.2 

1.71 

9 

0.121 

Using the drop box 

54.2 

46.9 

0.52 

9 

0.619 

Locating external links 

94.5 

96.0 

0.60 

9 

0.566 

Using the discussion boards 

74.1 

57.6 

1.52 

9 

0.163 

Using the calendarfunction 

90.0 

89.8 

0.07 

9 

0.945 

Taking quizzes 

84.7 

81.4 

1.12 

9 

0.294 

Checking your grade 

86.3 

80.2 

1.19 

8 

0.268 

Emailing the instructor 

93.9 

88.3 

1.25 

8 

0.247 


Cadets rated the second course they evaluated 
as slightly easier to use in nine of the 12 metrics 
examined. However, this difference was not 
statistically significant (Table 16). 

Table 16: Comparison of cadet (n = 29] mean ease of use scores 
for the web-enabled courses they evaluated first and 
second. Anchors for this scale ore 0 = very difficult 
12 = difficult 29 = somewhat difficult 66 = somewhat 
easy 82 = easy 100 = very easy. 


Task 

First 

Second 

t 

df 

P 

Locating the syllabus 

88.4 

92.6 

1.47 

28 

0.15 

Locating the announcements 

75.3 

77.2 

0.23 

24 

0.81 

Locating instructor information 

85.0 

86.0 

0.76 

28 

0.45 

Locating the course policies 

82.2 

84.9 

1.4 

28 

0.17 

Locating the assignments 

89.9 

91.6 

0.89 

28 

0.37 

Using the drop box 

49.3 

60.3 

141 

28 

0.16 

Locating external links 

87.1 

83.6 

0.68 

28 

0.5 

Using the discussion boards 

68.2 

64.5 

1.33 

28 

0.19 

Using the calendar function 

77.4 

67.7 

0.92 

22 

0.36 

Taking quizzes 

80.1 

81.9 

0.4 

26 

0.68 

Checking your grade 

69.7 

78.6 

0.94 

27 

0.35 

Emailing the instructor 

72.7 

81.9 

1.65 

28 

0.1 


2.3.7c Usefulness: 

There was no significant difference between the 
usefulness of the tasks cadets performed in 
courses built with Intralearn and WebCT for 11 of 
the 12 tasks, between courses built with 
Courselnfo and Intralearn for 9 of 12 tasks, or 
between courses built with Courselnfo and 
WebCT for 11 of 12 tasks (Tables 1 7a-c). As with 
the faculty the tasks that they were asked to 
evaluate were the same for each of the web- 
enabled courses. The cadets rated most of the 
features as somewhat useful to very useful. The 
only features that were rated neutral were using 
the drop box for submitting assignments to the 
instructor and the discussion boards for all 
courses. Cadets also rated checking their 
grades and using the calendaring function in 
Intralearn as neutral. 
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Table 17a: Comparison of cadet (n = 10) mean usefulness scores for 
tasks evaluated in web-enabled courses built with 
Intralearn and WebCT. Anchors for this scale are 0 = not 
at all useful, 12 = not very useful, 29 = of limited use, 66 = 
somewhat useful, 82 = useful, 100 = very useful. 


Task 

Intralearn 

WebCT 

t 

df 

P 

Locating the syllabus 

92.5 

95 

0.5 

9 

0.572 

Locating the announcements 

53.0 

68 

1.2 

6 

0.258 

Locating instructor information 

78 2 

79 

1.0 

9 

0.343 

Locating the course policies 

76.8 

76 

0.0 

9 

0.000 

Locating the assignments 

88.5 

91 

1.0 

9 

0.343 

Using the drop box 

36.8 

63 

2.3 

9 

0.042 

Locating external links 

65.1 

82 

1.5 

9 

0.152 

Using the discussion boards 

49.2 

52 

0.5 

9 

0.604 

Using the calendar function 

51.3 

85 

2.2 

7 

0.057 

Taking quizzes 

78.1 

85 

1.4 

7 

0.188 

Checking your grade 

55.6 

78 

1.7 

8 

0.120 

Emailing the instructor 

77.2 

67 

0.9 

9 

0.370 


Table 17b: Comparison of cadet (n = 9) mean usefulness scores for 
tasks evaluated in web-enabled courses built with 
Course!nfo (Cl) and Intralearn. Anchors for this scale are 
0 = not at all useful, 12 = not very useful, 29 = of limited 
use, 66 = somewhat useful, 82 = useful, 100 = very useful. 


Task 

Cl 

Intralearn 

t 

df 

P 

Locating the syllabus 

95.4 

89 7 

1.31 

8 

0.227 

Locating the announcements 

98.6 

56.3 

2.57 

7 

0.037 

Locating instructor information 

93.9 

856 

1.86 

8 

0.100 

Locating the course policies 

86.1 

82.3 

0.59 

8 

0.571 

Locating the assignments 

85.6 

894 

0.63 

8 

0.546 

Using the drop box 

80.6 

37.7 

2.86 

8 

0.021 

Locating external links 

97.1 

789 

2.11 

8 

0.067 

Using the discussion boards 

86.6 

84.2 

0.32 

8 

0.760 

Using the calendar function 

95.0 

33.4 

3.22 

4 

0.032 

Taking quizzes 

95.6 

90.0 

1.19 

8 

0.269 

Checking your grade 

84.6 

56.2 

1.39 

8 

0.201 

Emailing the instructor 

95.1 

72.3 

1.87 

8 

0.098 


Table 17c: Comparison of cadet(n = 10) mean usefulness scores for 
tasks evaluated in web-enabled courses built with 
Courselnfo (Cl) and WebCT. Anchors for this scale are 0 = 
not at all useful, 12 = not very useful, 29 = of limited use, 
66 = somewhat useful, 82 = useful, 100 = very useful. 


Task 

Cl 

WebCT 

t 

df 

P 

Locating the syllabus 

86.1 

89.9 

0.59 

9 

0.572 

Locating the announcements 

87.7 

87.7 

0.00 

9 

0.000 

Locating instructor information 

91.1 

88.9 

1.49 

9 

0.170 

Locating the course policies 

88.9 

85.9 

1.41 

9 

0.193 

Locating the assignments 

92.4 

88.7 

1.14 

9 

0.285 

Using the drop box 

57.7 

60.6 

0.35 

9 

0.733 

Locating external links 

96.5 

93.7 

1.43 

9 

0.187 

Using the discussion boards 

65.7 

55.8 

2.93 

9 

0.017 

Using the calendar function 

74.9 

86.7 

1.19 

9 

0.267 

Taking quizzes 

67.8 

73.7 

0.55 

9 

0.599 

Checking your grade 

82.0 

86.0 

0.35 

9 

0.733 

Emailing the instructor 

78.7 

74.7 

1.00 

9 

0.343 


2.3.7d Overall evaluation by cadets 


There was no significant difference between 
courses built with Intralearn and WebCT for 10 
statements designed to ascertain cadet's 
overall experience (Table 18a). Most of the 
statements were given a rating of neutral. 


Table 18a: Comparison of mean overall assessment scores by 
cadets (n = 10) for courses built with Intralearn and 
WebCT. Anchors for this scale are 0 = decidedly 
disagree, 25 = moderately disagree, 50 = neutral, 75 = 
moderately agree, 100 = decidedly agree. 


This software package: 

Intralearn 

WebCT 

t 

df 

P 

Took a reasonable amount of time 

62.0 

59 

0.3 

9 

0.749 

Had a good navigational layout 

48.5 

62 

0.8 

9 

0.398 

Had a logical layout 

57.0 

70 

0.8 

9 

0.394 

Had aesthetically pleasing screen displays 

63.0 

71 

0.5 

9 

0.618 

Was easy to use 

56.0 

67 

0.9 

9 

0.387 

Had an easy to use manual 

27.5 

52 

1.6 

1 

0.344 

Had instructive directions on the screen 

47.2 

47 

0.0 

9 

1.000 

Had useful features 

69.5 

79 

1.1 

9 

0.297 

Was intuitive in nature 

52.2 

72 

1.9 

8 

0.093 

Needs little training to use 

69.0 

68 

0.0 

9 

0.931 


The web-enabled course built with Courselnfo 
was rated significantly higher than the course 
built with Intralearn for 4 of the 10 statements 
used to assess the cadet's overall experience 
(Table 18b). Cadets rated 9 of the 10 statements 
as moderately agree to decidedly agree as they 
pertain to the Courselnfo course and only one of 
the statements was ranked as moderately agree 
for the Intralearn course. The other statements 
were rated as neutral. 

Table 18b: Pairwise comparison of overall assessment statements by 
cadets (n = 9) for courses built with Courselnfo (Cl) and 
Intralearn. Anchors for this scale are 0 = decidedly 
disagree, 25 = moderately disagree, 50 = neutral, 75 = 
moderately agree, 100 = decidedly agree. 


This software package: 

Cl 

Intralearn 

t 

df 

P 

Took a reasonable amount of time 

98.1 

71.5 

3.58 

7 

0.009 

Had a good navigational layout 

90.6 

45.6 

5.50 

7 

0.001 

Had a logical layout 

92.5 

55.4 

4.96 

7 

0.002 

Had aesthetically pleasing screen displays 

87.5 

72.5 

1.98 

7 

0.088 

Was easy to use 

89.9 

55.9 

3.72 

7 

0.007 

Had an easy to use manual 

73.8 

48.8 

1.89 

3 

0.155 

Had instructive directions on the screen 

70.0 

39.3 

2.33 

6 

0.058 

Had useful features 

95.6 

78.1 

2.15 

7 

0.068 

Was intuitive in nature 

93.8 

68.5 

3.56 

7 

0.009 

Needs little training to use 

83.8 

62.5 

1.66 

7 

0.141 


The course built with Courselnfo was rated 
significantly higher than the courses built with 
WebCT for 1 of the 10 statements used to assess 
the cadet's overall experience (Table 18c). 
However, the cadets rated 9 of the 10 
statements as moderately agree to decidedly 
agree as they pertain to Courselnfo and only 
one of the statements was ranked as 
moderately agree for WebCT. The other 
statements were rated as neutral. 
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Table 18c: Comparison of mean overall assessment scores by 
cadets (n = 10] for courses built with Courselnfo [Cl] and 
WebCT. Anchors for this scale are 0 = decidedly 
disagree, 25 = moderately disagree, 50 = neutral, 
75 = moderately agree, 100 = decidedly agree. 


This software package: 

Cl 

WebCT 

t 

df 

P 

Took a reasonable amount of time 

84.0 

72.5 

1.13 

9 

0.288 

Had a good navigational layout 

90.0 

73.0 

1.60 

9 

0.144 

Had a logical layout 

87.0 

77.0 

0.97 

9 

0.358 

Had aesthetically pleasing screen displays 

88.0 

61.0 

2.29 

9 

0.048 

Was easy to use 

86.0 

69.0 

1.63 

9 

0.136 

Had an easy to use manual 

91.0 

62.0 

1.99 

4 

0.118 

Had instructive directions on the screen 

60.0 

43.1 

1.86 

7 

0.105 

Had useful features 

89.8 

74.0 

1.57 

9 

0.151 

Was intuitive in nature 

82.5 

71.5 

0.91 

9 

0.385 

Needs little training to use 

81.0 

68.0 

1.07 

9 

0.311 


Cadets found information where they expected 
to find it less that 50% of the time in any of the 
courses built using any of the CMS products 
evaluated in this series of experiments (Table 19). 
Cadets also reported finding course documents 
and other information in places where it was not 
located. Similarly, when faculty were asked to 
place documents into a course using one of 
these products, different faculty members put 
the documents in different places (Table 20). 

Table 19: The locations within the course where cadets expected 
to find and did find the instructor's office hours and two 
course handouts in a web-enabled course built with one 
of three CMS products. The correct location of the 
documentor information is shown in bold face font. 



Syllabus 

Office location 

Honors document 

BB 

exp. 

found 

exp. 

found 

exp. 

found 

Course Information 

6 

13 

1 


7 

4 

Staff Information 



14 

19 



Course Documents 

3 




11 

14 

Other 

10(5) 

6(3) 

4(2) 


2(2) 









WebCT 







Instructor Information 



11 

20 



Course Documents 

11 

16 



15 

14 

Other 

9(3) 

5(3) 

9(4) 


5(2) 

6(2) 








Intralearn 







Syllabus 

8 

7 

9 

17 

4 

3 

Roster 



2 




Communication 



1 




Other 

_1M 

15(3) 

_5(3) 

_2(2) 

15(6) 

_ wsl 


Table 20: The locations where faculty placed two handouts into a 
course they were building using a CMS product. 



Syllabus 

Honors 

BB Courselnfo 



Course Information 

8 

4 

Course Documents 

6 

7 

Course Materials 


1 

Academics w/honors Clsrm pol 


1 




WebCT 



Course Documents 



Course Syllabus/Course Materials 

6 

3 

File Manager 

1 

1 

N/A 

3 

5 

Homepage 


1 




Intralearn 



Syllabus 



N/A 

3 

2 

Reference Materials 


1 

Uploads 

1 

1 


2.4 Discussion 

2.4.1 WWW experience by faculty and cadets 

The differences between the experience of 
faculty and cadets in the number of years they 
used and developed content for the www, was 
not significantly different, even though faculty 
are more educated and have more computer 
experience than cadets. Both groups over¬ 
estimated the number of years they used web 
browsers, as many reported using these tools 
before they had been developed (Anderberg, 
2001; Stewart, 2000). While it is possible that 
some participants may have used this 
technology in its early development and 
blended variousaspects of the internet with the 
www in their answers, it is more likely this over¬ 
reporting demonstrates that the use of the www 
is no longer a novel concept and both groups 
are very comfortable using this medium. 

Cadets reported using the web twice as many 
hours per week as faculty. While faculty reported 
more computer programming experience 
overall, cadets were just as familiar as faculty 
with authoring web content, and cadets were 
just as likely as faculty to host their own web site 
on a server located off the academy. However, it 
appears that neither group uses sophisticated 
development software or applications for their 
web sites, which is typical of most personal home 
pages on thewww(NetKontoret, 2001). 
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2.4.2 Selection of CMS features by both faculty and cadets 

When inexperienced participants were asked to 
determine what features were important and 
should be included in software to facilitate the 
development and use of web-enabled courses, 
both the faculty and cadets focused on their 
own use of the software and did not appear to 
consider the requirements of other user groups. 
Faculty did not rate features that were to be used 
by cadets such as finding grades or the ability to 
navigate the site higher than average. The 
faculty focused on their requirements for the 
tool, such as the uploading of class lists and the 
flexibility to add content from other resources. 
Both faculty and cadets rated the ability to 
support foreign languages as one of the least 
important features of these tools, yet the ability to 
support languages other than English would be 
essential for developing web-enabled 
curriculum for a foreign language course. The 2 
foreign language instructors and the 3 cadets 
enrolled in foreign language courses rated this 
feature as a "must-have". 

Cadets also focused on their own requirements, 
rather than the development of the course as a 
whole. They rated the exam question types they 
favored and theability to find materials as more 
important than other features. Cadets rated 
student tracking features, such as the ability to 
tell when and for how long astudent accessed a 
page as less important than most of the other 
featureslisted, even though these resources 
could be used by instructors to help determine 
which students were at risk of not being 
adequately prepared for anexam or class. Since 
neither group seems to have considered other 
user'srequirements, this illustrates the need to 
have all user groups involved in the selection 
process and development of requirements for 
these types of tools. 

Neither faculty nor cadets rated the 
collaboration tools, asynchronous or 
synchronous discussion groups, or email 
functions as being very important to include in 
the software packages. Unlike most colleges 


and universities, class attendance is mandatory 
for all cadets at USAFA, and faculty are generally 
available from 7:30 to 4:30 every day outside of 
class time. Therefore, the ability to have online 
discussions, or to collaborate on projects at a 
distance may not be as important as it is for 
those using these packages for true distance 
learning courses, where faculty and students are 
separated both temporally and spatially. In 
addition, all faculty and cadets have institutional 
email accounts, and all email addresses are 
listed in a centralized address book. Therefore, 
this feature within a course management 
package is redundant with a system already in 
place and in use at USAFA. 

2.4.3 Selection of preferred software package by 
faculty 

The faculty unanimously selected Blackboard 
Courselnfo as the preferred CMS package of the 
packages they tested for developing web- 
enabled courses. All of the faculty surveyed 
indicated that they chose this package 
because it was the easiest to use, and most 
cited ease of use as the single most important 
factor that helped them make their decision. 
They indicated that the navigational interface 
was intuitive, it required fewer mouse-clicks, and 
the steps necessary to load assignments and 
course documents were obvious in comparison 
to WebCT and Intralearn. 

Even though the tasks that they were asked to 
perform were the same, the faculty rated the 
course development tasks significantly more 
useful when using Courselnfo than with either 
WebCT or Intralearn. There was a strong statistical 
correlation between ease of use and usefulness 
scores for these tasks. If a task was easy to 
perform, the faculty tended to rate it as more 
useful to teaching and learning, and if the task 
was more difficult to perform, then the faculty 
rated it as less useful to teaching and learning. 
Therefore, even if WebCT and Intralearn had 
more useful tools than Courselnfo, they probably 
would still be rated as a less useful product 
overall. 
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One of the design features that made 
Courselnfo easier to use than WebCT and 
Intralearn was the layout of the menus that 
faculty encountered when using the software to 
build a course. The Courselnfo menu allows the 
user to view the different subheadings available 
and choose the desired function, whereas the 
WebCT interface only showed a few choices with 
each screen. In order to figure out what features 
might be available under a certain section, the 
faculty member had to click and wait for the next 
page to load. Intralearn also used a series of 
menus, however that package required the 
faculty member to upload all of their documents 
into the system first, and then go back and 
integrate it into the content. All of the faculty who 
tried to use Intralearn reported that once they 
uploaded a document into the system, they 
could not locate it. 

The faculty indicated that Courselnfo did not 
require a lot of training, whereas WebCT and 
Intralearn would require a lot of training to use. 
However, faculty did not seek out help or use the 
online manuals very often when using any of the 
software packages. One reason may be that 
these courses were not going to be used outside 
of this evaluation process. Therefore if tasks were 
difficult to understand or execute, faculty could 
skip that section and proceed to the next set. 
There was no difference in computer use or 
programming experience between the faculty 
who finished the experiments and the faculty 
who did not finish,which indicates that both 
groups had the necessary skills to finish the 
evaluation process. Most of the participants who 
did not finish, indicated that they were too busy, 
yet a few expressed a dislike of CMS tools overall 
and may have been biased because they did 
not like the first package that they tried. As one 
participant stated "the first software package 
was so distasteful, you could not convince me to 
try a second." 

Courselnfo also included six out of the seven 
features that faculty rated as more important 
before they began evaluating the software 
packages. The only feature that was not present 


was the ability to upload questions as a question 
file, and they had not been asked to do this task 
as part of the evaluation process. Finding 
features that they had predetermined as useful 
may have validated their beliefs that this product 
was beneficial to teaching. 

2.4.4 Selection of preferred software package by 
cadets 

Cadets who compared courses prepared with 
Courselnfo to Intralearn, unanimously chose the 
course designed with Courselnfo. Although three 
of the cadets preferred the course designed with 
WebCT to the Courselnfo course, the other seven 
preferred the course designed with Courselnfo. 
Cadets were equally divided when given a 
choice between courses designed with WebCT 
and Intralearn. 

Overall the cadets found all three packages 
easy to use, and only 34% of the cadets used the 
online manuals or asked for help during the 
process. Cadets did indicate that the second 
package that they tried was easier to use on 
average than the first package that they tried. 
This was probably a function of habituating to the 
tasks and to the worksheets, than the actual 
software packages. The software was assigned 
to the cadets in a randomized fashion so that 
each software package had an equal 
probability of being first or second. However, this 
was not in and of itself a deciding factor since 
cadets did not show a statistical bias towards 
choosing the second course as their overall 
preferred choice. 

The cadets rated all of the packages with 
equivalent scores for the amount of training that 
they would require. The tasks that cadets were 
assigned did not take as much time as those 
assigned to the faculty. Instead of having to 
developweb-enabled courses with the software, 
they just had to navigate courses that were 
already developed. Cadets are accustomed to 
navigating web sites, so these tasks were 
probably not as novel to them as developing 
web-enabled courses were to the faculty. Many 
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websites have unintuitive navigational layouts, 
and "hunt and click" navigation is frequently used 
by people who browse the web (Nielsen, 1998). 

For all of the courses, the course documents 
were not where the cadets expected to find 
them, which increased the amount of time the 
cadets had to search to find the appropriate 
handout. The navigation schemes of the 
courses did differ with respect to layout of the 
assignments. Using the layouts of Courselnfo and 
WebCT, cadets would have to search under 
different headings to find all the handouts and 
assignments that might be due in a particular 
lesson. For thelntralearn course, the layout was 
ordered lesson by lesson. Of the cadets that 
chose Intralearn over WebCT, all of them 
reported the navigational layout as the defining 
reason. This preference did not carry over into 
the Courselnfo tests. Even though some of the 
cadets preferred this navigational layout of the 
Intralearn course, the Intralearn course received 
a lower score for being intuitive than both the 
courses built in Courselnfo and WebCT. 

Of the cadets who preferred WebCT and 
Courselnfo to Intralearn, they cited additional 
features available in both WebCT and 
Courselnfo, such as an integrated calendar and 
the ability to find their grades. Although cadets 
took quizzes and were asked to check their 
grades using all three packages, most of the 
cadets were unable to find their grades in the 
Intralearn course. 

Cadets also rated the features in the courses 
such as asynchronous and synchronous 
discussion groups, integrated calendar, using 
external links, taking quizzes and finding their 
grades as useful. The only feature that cadets did 
not rate as useful was the drop box for submitting 
assignments to their instructors. Many of the 
cadets commented that this was redundant to 
submitting assignments as email attachments, 
which is the current system. Several cadets also 
commented that using a separate calendar for 
courses was confusing, and that they would 
prefer to have all of their assignments and 
announcements integrated into one 


calendaring system, which is part of the 
enterprise system we seek to develop and 
deploy at USAFA. 

As with the faculty tests, features that the cadets 
rated as being more important than the rest 
were included in the courses that they favored. 
However, since they were not taking a real 
course, the grades that were posted were few 
and of limited importance. In addition, instructor 
feedback was not provided with the grades that 
were issued during the evaluation process. As 
with faculty, finding features that they had 
predetermined as useful may have validated 
their beliefs that using these products to web- 
enable courses was useful. 

Section 3: Experienced USAFA Faculty and Cadets 
Rate Blackboard Courselnfo 

3.1 INTRODUCTION 

In section 2 of this study, both faculty and cadets evaluated 
CMS products using a discrete set of criteria. However, 
performing a task in a simulated environment is not always 
the same as using the tool on a daily basis. Since the faculty 
members and cadets are more experienced at using the 
tool after a semester than during the initial choice phase, 
they should provide finer discrimination between ease of 
use and the usefulness of the different features. In addition, 
faculty members teaching different disciplines are likely to 
use the tool in slightly different ways, and put documents in 
different areas providing variety even within a standard 
template. Using the tools for an entire semester can 
provide us data as to the frequency of use of some of the 
features packaged within the products, providing 
additional data on the importance of certain features. 

Therefore, the purpose of this part of the study was to have 
faculty members and cadets provide final feedback on 
using Course Management System software and how it 
functioned to support the curriculum at USAFA after they 
had been using for the tool set for a semester. Since all of 
the faculty members preferred Courselnfo in the choice 
experiments, this was the only CMS analyzed during this 
phase of the study. However, the data collected on 
features used and the faculty's ability to use Courselnfo to 
web-enable curriculum at USAFA, helps to structure a list of 
requirements for the role of CMS software in producing an 
online educational environment. 
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3.2 METHODS 

3.2.1 Instructor evaluations 

Thirteen Faculty members from the United States 
Air Force Academy (USAFA) who had been using 
Courselnfo to develop their online supplemental 
course materialsfor Fall '00 volunteered to answer 
the questionnaires in December 2000. Some of 
these faculty had participated in the preliminary 
choice experiments outlined above, but others 
joined the study after the completion of that 
phase. As with the first phase, these faculty 
members were asked to rate each of the features 
listed in Table 1 according to how important they 
were to producing web-enabled courses. Faculty 
were then asked how often they used each of the 
features available in Courselnfo and to rate them 
according to ease of use and usefulness using a 
1-100 visual analog scale (Miller 1999). They were 
also given a series of questions designed to assess 
their overall satisfaction using the software. 

3.2.2 Cadet evaluations 

Cadets evaluated the course in which they were 
enrolled. They were asked to report how often 
they used each of the features included in 
Blackboard Courselnfo, and to report the ease of 
use and usefulness of the product on a 1-100 
visual analog scale (Miller, 1999). Cadets also 
answered a set of questions designed to assess 
their overall like or dislike of Blackboard 
Courselnfo as a CMS, rather than their like and/or 
dislike of the content material of the course itself. 
The responses of the cadets were kept 
anonymous, and pooled for all courses. 

3.2.3 Data collection and analysis 

Data for both the faculty and cadets were 
collected by self-report on surveys and entered 
into a MS Excel spreadsheet. Summary statistics 
were calculated using MSExcel and are reported 
as mean + sd unless otherwise noted. Data were 
imported into SPSS for further analysis. Faculty 
course questionnaire data was obtained from 
the Center for Educational Excellence. All 
identifiers were stripped from the data so that the 
identity of the individual faculty members 
remained anonymous. 


Blackboard Courselnfo was housed on a Sun 
Ultra Sparc 5 server located in the Institute for 
Information Technology Applications laboratory 
in Fairchild Hall at USAFA. The web pages were 
accessed through the academy's intranet. 
Cadets and faculty were instructed to bypass 
the proxies for efficiency and speed. 

3.3 RESULTS 

3.3.1 Faculty participants 

There were 10 male and 3 female faculty 
members representing departments of various 
disciplines including Biology, Engineering 
Mechanics, Economics and Behavioral 
Sciences. Seven of the participants (54%) had 
created home pages for their classes in previous 
semesters, but for six (46%) of the participants this 
was their first semester creating web pages and 
using the web as a teaching tool. Only two (15%) 
of the faculty maintained additional web pages 
not included in the Courselnfo CMS at USAFA, 
and only 23% maintained a web-site with an 
internet service provider outside USAFA. Seven 
(54%) of the faculty participants chose to use 
Blackboard this semester because they were 
part of the original choice experiments (Section 
2 of this report), whereas six (46%) of the faculty 
chose to use the software upon the 
recommendation of the course director 
overseeing the syllabus for their course. 

3.3.2 Cadet participants 

Fourteen male and 5 female cadets evaluated 
the online resources ofthe faculty members who 
were using Courselnfo during the Fall '00 
semester including courses in Biology (n=10). 
Physics (n=3). Economics (n=2). Engineering 
Mechanics (n = 2), and Leadership (n = l). 
Because the instructors using Courselnfo were 
teaching courses targeted to that level, the 
majority (15 or 79%) of the cadets who 
evaluated the online materials were third-class 
(sophomore) cadets, one first-class (senior) 
cadet and 2 second-class (junior) cadets. Only 8 
(41.2%) of the cadets had used web-sites 
created by their instructors to supplement the 
classroom curriculum, whereas most of the 
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cadets (n =11, 57.8%) were using this resource 
for the first time. None of the cadets had ever 
used online course materials authored using 
Courselnfo CMS software or other CMS 
packages in previous semesters. 

3.3.3 Rankings of selection criteria by faculty and 
cadets 

When experienced faculty and cadets were 
asked to rank features used to evaluate the 
course management software packages, both 
groups tended to give most of the features 
similar scores. The Grand mean rating given to 
the features by faculty was 4.06 + 0.88 for 
faculty and 4.21 + 0.58 for cadets on a 6 point 
scale where 1 indicated that the feature was of 
very little importance, and 6 was very important 
or a "must have". 

Ten features were rated 5.0 or higher and 
considered to be more important than the 
others to faculty developing curriculum. Most of 
these features were course administration 
features, the ability to author content without the 
knowledge of HTML, grading and quizzing 
features (Table 21). Seven features received a 
score of 3.8 or below and were considered less 
important including the ability to support foreign 
languages, the use of discussion groups, video¬ 
teleconferencing, virtual field trips and the type 
of operating system. The ability to conform to 
external standardization such as IMS 
specifications received a rating of 2.0 and was 
designated as least important. 

Table 21: Ratings of course management software features that 
faculty (n = 13] rated significantly higher or lower than the 
grand mean for all features. The corresponding scores 
given by cadets [n=19] are also shown with features that 
were given equivalent ratings by both groups shown in 
bold. The ratings were on a 6 point scale where 1 
indicated that the feature was of very little importance to 
web-enabled curriculum, and 6 was very important or a 
"must have". The grand mean score for all features was 
4.06 + 0.88 for faculty and 4.21 + 0.58 for cadets. 


Feature 

Faculty 

sd 

Cadet 

sd 

more important: 





Does not require knowledge of HTML 

5.5 

0.1 

4.9 

1.2 

Class lists can be presented, saved, and printed using Excel 

5.5 

0.7 

4.1 

1.5 

Student access to progress data available 

5.4 

1.4 

5.3 

0.9 

Class list can be uploaded as a file 

5.4 

0.9 

4.0 

1.5 

Table of contents 

5.1 

1.0 

5.0 

1.1 

Course can be downloaded for safekeeping 

5.1 

1.2 

4.1 

1.7 

Scores can be exported into Excel 

5.1 

1.7 

5.0 

1.0 

Quizzes automatically graded and entered into gradebook 

5.1 

1.7 

5.1 

1.1 

Online instructor manual 

5.0 

1.3 

4.2 

1.6 

Can use a mixture of question types on a single quiz 

5.0 

1.5 

5.0 

0.9 

less important: 





Can Support Foreign Languages 

2.9 

1.8 

3.7 

1.8 

Virtual field trips within discussion pages 

2.7 

1.6 

3.6 

1.9 

Logged synchronous discussion 

2.5 

1.3 

3.2 

1.7 

Shared whiteboard 

2.5 

1.4 

3.5 

1.8 

Adaptable for desktop videoteleconferencing 

2.4 

1.5 

3.8 

1.8 

least important 





Searchable asynchronous discussion 

2.3 

1.2 

3.3 

1.7 

Operating System 

2.3 

5.1 

3.3 

1.7 

Conforms to Instruction Management System specs 

2.0 

0.7 

3.5 

1.5 


Twelve features were rated 4.8 or higher and 
considered to be more important than the 
others to cadets for web-enabled curriculum. 
The features included in this group were the 
ability to access grades, several quizzing 
features and the ability to produce web 
resources without knowing how to author HTML 
(Table 22). Thirteen features received a score of 
3.6 or below and were considered least 
important including the ability to conform to 
external web development specifications such 
as IMS, email features, the ability to track how 
often and when a student accessed the online 
materials, asynchronous and synchronous 
discussion pages and the operating system 
used to run the software (Table 22). 

Table 22: Ratings of course management software features that 
cadets (n=19) rated significantly higher or lower than the 
grand mean for all features. The corresponding scores 
given by faculty (n= 13] are also shown with features that 
were given equivalent ratings by both groups shown in 
bold. The ratings were on a 6 point scale where 1 
indicated that the feature was of very little importance to 
web-enabled curriculum, and 6 was very important or a 
"must have". The grand mean score for all features was 
4.06 + 0.88 for faculty and 4.21 + 0.58 for cadets. 
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Feature 

Cadet 

s.d. 

Faculty 

s.d. 

more important: 





Ability to add offline grades 

5.3 

0.8 

4.8 

1.7 

Make quizzes with true-false/multiple choice questions 

5.3 

0.8 

4.9 

1.7 

Student access to progress data available 

5.3 

0.9 

5.4 

1.4 

One question at-a-time testinq capability 

5.3 

2.5 

4.6 

1.7 

Instructor comments available with grade 

5.2 

1.0 

4.2 

1.6 

Quizzes automatically graded and entered into gradebook 

5.1 

1.1 

5.1 

1.7 

Grade statistics and/or histograms 

5.0 

1.2 

4.7 

1.B 

Scores can be exported into Excel 

5.0 

1.0 

5.1 

1.7 

Table of contents 

5.0 

1.1 

5.1 

1.0 

Can use a mixture of question types on a single quiz 

5.0 

0.9 

5.0 

1.5 

Does not require knowledge of HTML 

4.9 

1.2 

5.5 

0.1 

Fill in the blank 

4.8 

1.2 

4.7 

1.7 

less important: 





Conforms to World Wide Web Consortium specs 

3.6 

1.5 

3.3 

0.9 

Collaborative work area for group 

3.6 

1.6 

3.4 

1.5 

One to one course email 

3.6 

1.6 

3.7 

1.7 

Can track how often student accesses pages 

3.6 

1.5 

3.9 

1.0 

Virtual Field trips within disscussion pages 

3.6 

1.9 

2.7 

1.6 

One to many course email 

3.6 

1.5 

4.0 

1.7 

Shared whiteboard 

3.5 

1.8 

2.5 

1.4 

Conforms to IMS specifications 

3.5 

1.5 

2.0 

0.7 

Group presentation area 

3.5 

1.7 

3.2 

1.6 

Individual presentation area/homepage 

3.4 

1.5 

3.3 

1.0 

Searchable asynchronous discussion 

3.3 

1.7 

2.3 

1.2 

Operating system 

3.3 

1.7 

2.3 

5.1 

Logged synchronous discussion 

3.2 

1.7 

2.5 

1.3 


3.3.4 Faculty evaluation ofCourselnfo 


Most of the faculty members did not use all of the 
features that were available in Courselnfo. The 
most widely used features included adding 
assignments and documents, adding external 
links, creating course announcements and the 
online grade book. Some of the faculty used the 
online quizzing, student tracking, emailing and 
calendar features, and one or none of the faculty 
used the discussion boards, chat features or any of 
the collaborative work group features (Table 23). 

Table 23: The number of faculty that used the features and the number 
of times that they used each feature. N = the number of 
faculty that used the feature,andMean = the number of 
time those faculty used the feature in a semester. 


Frequency using the following features: 

N 

Mean 

s.d. 

Range 

Student tracking 

7 

64.3 

58.7 

18-180 

Online quizzes 

8 

25.9 

14.8 

9-54 

Adding assignments/documents 

11 

23.7 

10.1 

9-36 

Asynchronous discussions boards 

1 

36.0 



Chat feature 

1 

36.0 



Adding external links 

10 

22.5 

12.9 

9-54 

Online grade book 

9 

70.0 

82.7 

18-270 

Creating announcements 

10 

32.4 

20.4 

18-72 

Use BB to send email to the class 

5 

27.0 

25.5 

9-72 

File sharing 

1 

54.0 



Drop box 

1 

36.0 



Group workspace 

0 




Calendar 

5 

25.2 

9.9 

18-36 

Other 

0 





Of the faculty who used those features, they rated 
course set up, creating announcements, sending 
email to the class and grade book functions as 
easy to very easy. They rated eight of the other 
features as somewhat easy to easy, including 
adding assignments and uploading web pages, 
composing quizzes and using the calendar. The 
other features were rated as neutral, or were not 
used by the participants (Table 24). 


Ta b I e 24: Experienced faculty (n = 13] mean ease of use scores for 

Courselnfo (Cl). Anchors for this scale are 0 = very 
difficult 12 = difficult 29 = somewhat difficult 66 = 
somewhat easy 82 = easy 100 = very easy. N = the 
number of faculty that used the feature, and Mean = the 
average rating assigned to that feature. 


Task 

N 

Mean 

s.d. 

Course set up (uploading syllabus and policies) 

10 

82.7 

13.0 

Adding assignments 

8 

66.3 

27.4 

Uploading web pages 

7 

74.7 

12.4 

Linking other web pages 

8 

78.9 

14.0 

Composing quizzes 

7 

75.4 

19.4 

Administrative functions overall 

8 

77.1 

13.9 

Enrolling cadets 

8 

61.4 

29.4 

Creating announcements 

11 

88.5 

11.1 

Sending email to class 

7 

95.0 

6.5 

Setting up grade book 

8 

87.4 

8.4 

Entering grades into grade book 

10 

738 

27.8 

Exporting gradebook into excell 

6 

84.8 

9.9 

Discussion groups 

1 

41.0 

58.0 

File sharing 

1 

32.5 

46.0 

Group workspace 

0 



Calendar 

5 

77 6 

22.5 

Chat group 

0 



Modifying look and feel of course 

7 

75.9 

15.8 


As with the original choice experiments, faculty 
rated those features that were easy to use as the 
most useful. Of the faculty who used those 
features, they rated six features as useful to very 
useful including course set up, adding 
assignments, uploading web pages, and 
creating announcements (Table 25). They rated 
five of the features as somewhat useful to useful, 
including grade book functions, enrolling cadets, 
administrative functions and composing quizzes. 
The other features were rated as neutral. 

Table 25: Comparison of mean usefulness scores given by faculty 
(n = 13] for Courselnfo (Cl). Anchors for this scale are 0 = 
not at all useful, 12 = not very useful, 29 = of limited use, 
66 = somewhat useful, 82 = useful, 100 = very useful. N 
= the number of faculty that used the feature, and Mean 
is the average rating assigned to that feature. 


Task 

N 

Mean 

s.d. 

Course set up (uploading syllabus and policies) 

10 

89.6 

10.5 

Adding assignments 

7 

91.4 

18.6 

Uploading web pages 

7 

91.4 

14.6 

Linking other web pages 

7 

88.9 

14.5 

Composing quizzes 

8 

82.5 

18.0 

Administrative functions overall 

9 

78.2 

18.0 

Enrolling cadets 

10 

77.5 

19.3 

Creating announcements 

11 

84.4 

16.7 

Sending email to class 

8 

56.0 

27.2 

Setting up grade book 

9 

71.8 

28.7 

Entering grades into grade book 

11 

70.5 

27.4 

Exporting gradebook into excell 

7 

75.1 

18.4 

Discussion groups 

5 

44.0 

34.4 

File sharing 

5 

46.4 

24.7 

Group workspace 

5 

46.4 

27.7 

Calendar 

9 

641 

26.6 

Chat group 

5 

46.4 

27.7 

Modifying look and feel of course 

9 

58.2 

29.0 
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Nine of the 13 faculty indicated that it took them 
the same amount of time to prepare for their 
course using Courselnfo and only three faculty 
members reported that it took them more time 
to prepare for class using the CMS. When asked 
to estimate the payoff in future class 
preparations, four (38.5%) of the faculty 
indicated that it would take them less time to 
prepare for class in the future, now that they had 
some of the resources online. However, ten 
(61.5%) felt that it would take them the same 
amount of time to prepare for future classes. 
Nine (69.2%) of the faculty indicated they would 
use Courselnfo again for the spring semester, 
whereas 3 (30.8%) indicated that they would not 
be using the software product. This is in part due 
to the preferences of the course director. 

For 10 statements designed to ascertain their 
overall experience using Courselnfo, faculty 
moderately agreed with eight out of ten of the 
statements. Courselnfo had a good 
navigational and pedagogical layout, was easy 
to use, required little training to use, as well as 
had pedagogically useful features. The faculty 
scored two of the 10 statements as neutral 
including took a reasonable amount of time to 
accomplish tasks and had an easy to use 
instructor's manual (Table 26). 

Table 26: Comparison of mean overall assessment scores by faculty 
(n = 13) for Courselnfo. Anchors for this scale ore 
0 = decidedly disagree, 25 = moderately disagree, 
50 = neutral, 75 = moderate^ agree, 100 = decidedlyagree. 


This software package: 

N 

Mean 

s.d. 

Took a reasonable amount of time to accomplish tasks 

12 

68.8 

26.4 

Had a good navigational layout 

13 

80.8 

11.3 

Had a good pedagogical layout 

10 

74.5 

17.6 

Had aesthetically pleasing screen displays 

13 

76.2 

22.2 

Was easy to use 

13 

78.1 

15.9 

Had an easy to use user’s manual 

7 

65.0 

21.0 

Had instructive directions on the screen 

12 

75.0 

25.2 

Had pedagogically useful features 

11 

75.9 

16.9 

Was intuitive in nature 

12 

73.3 

21.0 

Needs little training to use 

13 

72.3 

21.3 


3.3.5 Cadet Evaluation of Courselnfo 

Most of the cadets did not use all of the features 
that were available in Courselnfo. The most 
widely used features included taking online 
quizzes, reading announcements and checking 
the online grade book. Few of the cadets 
accessed the online assignments and 


documents, used the email function or the 
online calendar function, while two to none of 
the cadets used the discussion boards, chat 
features or any of the collaborative work group 
features (Table 27). 

Table 27: The number of cadets that used the features and the 
number of times that they used each feature. N = the 
number of faculty that used the feature, and mean 
indicates the number of time those faculty used the 
feature in a semester. 


Frequency using the following features: 

N 

Mean 

s.d. 

range 

Online quizzes 

15 

18.0 

6.1 

9-36 

Adding assignments/documents 

4 

27.0 

9.0 

18-36 

Asynchronous discussion boards 

2 

22.5 

6.4 

18-27 

Chat features 

0 




Adding external links 

2 

18.0 



Online grade book 

13 

22.8 

12.0 

9-54 

Reading announcements 

13 

24.2 

13.4 

18-54 

Use BB to send email to your instructor 

4 

27.0 

10.3 

18-36 

File sharing 

0 




Drop box 

1 

36.0 



Group workspace 

0 




Calendar 

5 

28.8 

16.1 

18-54 


Of the cadets who used those features, they 
rated eight of the twelve features as easy to very 
easy to use including locating announcements 
and course documents, sending email to the 
class and checking their grade. They rated the 
other features as somewhat easy to easy, 
including turning in an assignment using the 
drop box, using the discussion boards, and using 
the calendar (Table 28). 

Table 28: Experienced cadet (n = 19] mean ease of use scores for 
Courselnfo (Cl). Anchors for this scale are 0 = very 
difficult, 12 = difficult, 29 = somewhat difficult, 66 = 
somewhat easy, 82 = easy, 100 = very easy. N = the 
number of cadets who rated the feature, and Mean = 
the average rating assigned to that feature. 


Task 

N 

Mean 

s.d. 

Locating the syllabus 

16 

80.9 

20.7 

Locating the announcements 

18 

86.8 

19.5 

Locating instructor information 

18 

85.4 

19.2 

Locating the course policies 

17 

83.6 

19.2 

Locating the assignments 

19 

88.9 

14.8 

Turning in an assignment using the drop box 

8 

62.4 

37.9 

Locating the external links 

14 

83.4 

17.5 

Using the discussion boards 

7 

65,6 

35.7 

Using the calendar function 

10 

69.9 

35.4 

Taking quizzes 

16 

91.1 

14.8 

Checking your grade 

16 

87.4 

15.0 

Emailing the instructor 

8 

81.3 

18.4 


The only Courselnfo feature that the cadets rated 
as useful to very useful was checking their grade 
(Table 29). They rated seven of the features as 
somewhat useful to useful including locating the 
syllabus, announcements and assignments, 
taking quizzes and emailing the instructor. The 
other features were rated as neutral. 
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Table 29: Comparison of mean usefulness scores given by cadets 
(n = 19) for Course!nfo (Cl). Anchors for this scale are 
0 = not at all useful, 12 = not very useful, 29 = of limited 
use, 66 = somewhat useful, 82 = useful, 100 = very 
useful. N = the number of cadets who rated the feature, 
and Mean is the average rating assigned to that feature. 


Task 

N 

Mean 

s.d. 

Locating the syllabus 

15 

63.7 

32.1 

Location the announcements 

16 

70.6 

28.6 

Locating instructor information 

16 

68.9 

28.1 

Locating the course policies 

15 

58.0 

34.4 

Locating the assignments 

17 

77.8 

28.1 

Turning in an assignment using the drop box 

9 

39.1 

47.4 

Locating the external links 

10 

63.8 

34.6 

Using the discussion boards 

9 

40.2 

39.9 

Using the calendar function 

10 

56.6 

41.7 

Taking quizzes 

15 

76.7 

33.8 

Checking your grade 

15 

85.4 

25.6 

Emailing the instructor 

9 

62.9 

38.7 


Thirteen of the 19 cadets indicated that it took 
them the same amount of time to prepare for 
their course when the course was supplemented 
with materials prepared using Courselnfo when 
compared to other courses taught with 
nonCourselnfo web sites and four cadets 
indicated that it took them up to four hours a 
week less. Only 2 cadets reported spending up 
to 1.5 hours more each week in class 
preparation. Fourteen (73.6%) of the cadets 
indicated they would like to use Courselnfo 
again during the spring semester, whereas five 
(26.4%) indicated that they would rather not use 
web sites produced by the software product. 

For 10 statements designed to ascertain their 
overall experience using Courselnfo, cadets 
moderately agreed with eight out of ten of the 
statements. Courselnfo had a good 
navigational and logical layout, was easy to use, 
required little training to use, as well as had useful 
features. The cadets scored two of the 10 
statements as neutral including took a 
reasonable amount of time to accomplish tasks 
and had an easy to use manual (Table 30). 


This software package: 

N 

Mean 

s.d. 

Took a reasonable amount of time to accomplish tasks 

18 

66.1 

30.3 

Had a good navigational layout 

19 

75.7 

24.6 

Had a logical layout 

19 

72.3 

24.5 

Had aesthetically pleasing screen displays 

19 

83.8 

13.4 

Was easy to use 

19 

80.5 

23.4 

Had an easy to use user’s manual 

16 

54.6 

34.7 

Had instructive directions on the screen 

18 

73.3 

25.5 

Had useful features 

19 

73.1 

24.8 

Was intuitive in nature 

19 

72.1 

26.4 

Needs little training to use 

19 

77.3 

29.0 


Table 30: Comparison of mean overall assessment scores by 
cadets (n = 19] for Courselnfo. Anchors for this scale are 
0 = decidedly disagree, 25 = moderately disagree, 
50 = neutral, 75 = moderately agree, 100 = decidedly 
agree. 

3.3.6 End of course assessment data 

Faculty members who used Courselnfo 
received slightly higher course evaluations than 
other faculty members during the Fall '00 
semester (Figure 1) for ten questions chosen to 
assess teaching performance. However, those 
thirteen faculty members also received slightly 
higher course evaluations than the rest of the 
faculty in the semester before they began to use 
Courselnfo. In one of the core courses, half of the 
faculty used Courselnfo as a supplemental 
teaching tool and the other faculty did not use 
web pages to supplement their teaching. 
Faculty members who were using Courselnfo 
received higher end of course assessments than 
faculty who were not for that course (Figure 2). 

Figure 1: The end of course assessment data for the thirteen 
Courselnfo users during the semester they were using the 
product compared with all faculty during the same 
semester and the semester before they were using the 
product. The ratings were based on a 1 to 6 scale with 1 
equal to very poor and 6 equal to excellent. None of the 
scores were statistically significant from the other scores 
at the 0.05 level. 
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Question: 

I. Instructor's ability to stimulate my interest was: 

3. Instructor's ability to provide clear, well-organized 
instruction was: 

7. Instructor's knowledge of course material was: 

8. As a military role model or civilian professional role 
model, my instructor was: 

II. Availability of extra help when needed was: 

13. Course organization was: 

14. Clarity of course objectives and requirements was: 

19. Quality and usefulness of course text(s) were: 

20. The course as a whole was: 

23. The instructor's effectiveness in facilitating my learning 
in the course was: 

Figure 2: Comparison of end of course assessment data for 4 core 
course instructors who used Blackboard Courselnfo 
compared with 4 instructors teaching the same course 
who did not use the software or other web site 
supplementation. The ratings were based on a 1 to 6 
scale with 1 equal to very poor and 6 equal to excellent. 
The differences between the scores for questions 8, 14, 
and 19 were statistically significant at the 0.05 level. 



3. Instructor's ability to provide clear, well-organized 
instruction was: 

7. Instructor's knowledge of course material was: 

8. As a military role model or civilian professional role 
model, my instructor was: 

11. Availability of extra help when needed was: 

13. Course organization was: 

14. Clarity of course objectives and requirements was: 

19. Quality and usefulness of course text(s) were: 

20. The course as a whole was: 

23. The instructor's effectiveness in facilitating my learning 
in the course was: 


3.4 DISCUSSION 

3.4.1 Faculty and cadet rankings of selection criteria 

When participants who had been using 
Courselnfo for one semester were asked to 
determine which features were important and 
should be included in software to facilitate the 
development and use of web-enabled courses, 
both faculty members and cadets followed the 
trend of participants from the choice 
experiments and focused on their own 
requirements. Cadets focused on the types of 
quiz questions that would appear on exams and 
the ability to check their scores online, and the 
faculty continued to focus on administrative 
procedures and quiz construction. However, 
there was more agreement between these two 
groups than from the previous experiments. Both 
groups felt that student access to progress data, 
automatically graded quizzes, using a mixture of 
question types on a single quiz, table of contents 
and the ability to create pages without the 
knowledge of HTML were more important than 
other features required of a CMS product. 

The faculty members and the cadets continued 
to rate the discussion groups, shared 
whiteboards, virtual field trips and the type of 
operating system on which the CMS will operate 
as features that are not necessary, or less 
important than most of the other features. In 
addition, faculty continued to rate the support of 
foreign languages as less important, although 
there were no foreign language instructors 
represented in this population. The importance 
of a CMS to support foreign language instruction 
has been indicated in a separate 
communication as a requirement by the foreign 
language department. 

Previously, cadets rated all student tracking 
features, including how often a student 
accesses pages, time spent on each page, and 
when a student accesses pages as less 
important than other features. However, cadets 
who were enrolled in courses using Courselnfo 
did not list these features as less important. 
Cadets who had not used the system prior to 
rating the features may have been concerned 


l-manager’s Journal of Educational Technology, Vol. 2 • No. 1 • April-June 2005 97 





























































RESEARCH PAPERS 


how faculty would use this information, but 
cadets enrolled in courses using Courselnfo did 
not seem to object to the way this information 
was used by those faculty members who chose 
to access student tracking features. 

3.4.2. Experienced faculty ratings of Blackboard 
Courselnfo 

Most of the faculty used Courselnfo to post 
documents and assignments, to send their 
cadets announcements, add external links, to 
report scores and to administer online quizzes. Of 
these commonly used tasks, the faculty rated 
most of them as somewhat easy to easy to use, 
as well as somewhat to very useful. When asked 
what they most liked about using Courselnfo, 
most of the faculty reported the use of 
automatically graded quizzes, cadet access to 
grade book, and a central place to post 
announcements and assignments. Although 
faculty used the online grade book, many had 
problems using it as designed and found it more 
cumbersome to enter grades into the program 
than into a spreadsheet such as MSExcel. 

Very few of the faculty used the collaborative 
features of Courselnfo including the discussion 
boards, chat feature, file sharing, drop box or 
group workspace. Of those who rated those 
features, they were rated as neutral with respect 
to ease of use and usefulness. When asked what 
feature they would delete, ten (76.9%) indicated 
that they would not delete any of the 
componentsand only three (23.1%) said they 
would delete any of the features, naming the 
chat and discussion boards specifically. As 
stated in an earlier section of this report, that may 
be influenced by our use of the product to 
supplement traditional classroom education, 
rather than to deliver distance or online courses. 

Most faculty members were satisfied with the 
product overall and moderately agreed that it 
had a good pedagogical layout, 
pedagogically useful features, and that it did not 
take them more time to prepare for class than to 
prepare for a similar course without Courselnfo 
web-enabled curriculum materials. Many 
faculty members indicated that it lacked some 


key administrative features, for example there 
was no way to divide grade books, 
announcements or documents into multiple 
sections of the same course without duplicating 
the entire course. Similarly, it was not possible to 
share calendars, quiz questions, handouts, or 
content among instructors teaching the same 
course. Therefore, this software seems best 
suited for providing web-enabled instructional 
support to a faculty member who is teaching 
one section of a single course, and who does 
not share large amounts of content with 
instructors teaching other sections. 

3.4.3. Experienced cadet ratings of Blackboard 
Courselnfo 

Most of the cadets used Courselnfo to take 
quizzes, read announcements and access their 
grades. Of these commonly used tasks, cadets 
rated most of them as easy to very easy to use, 
as well as somewhat useful to useful. When 
asked what they liked most about using 
Courselnfo, most of the cadets reported the use 
of automatically graded quizzes with instant 
feedback, access to grade book, a central 
location for all materials and course information, 
and that misplaced handouts or ones not 
picked up in class could be printed out off the 
web site. One cadet also commented that using 
Courselnfo also wasted less paper. 

Very few or none of the cadets used the 
interactive features of Courselnfo including the 
discussion boards, chat feature, file sharing, 
drop box or group workspace, presumably 
because they were not assigned by their 
instructor. Of those who rated those features, 
they rated them as somewhat easy to use and 
neutral with respect to usefulness. When asked 
what feature they would delete, only three 
(15.8%) indicated that they would delete 
anything; two of these cadets would delete the 
chat and discussion boards, and another cadet 
commented that they would delete the campus 
and community center pages (these sections 
were not part of the this test, and were therefore 
not populated with events). 
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While most cadets were satisfied with the 
product overall, and moderately agreed that it 
was easy to use, had a good logical layout, 
useful features, and that it did not take them 
more time to prepare for class than not using the 
web-based support materials, many cadets 
expressed disappointment that the faculty 
members were not using the software to its fullest 
extent. Some indicated that they would like to 
see more than just class notes and 
announcements posted. Cadets expressed 
interest in online quizzes and answers, and others 
expressed interest in the chat features. Although 
some of their requests may have been 
unrealistic ("I really don't like homework so if 
Blackboard could stop it, that would be nice"), 
this indicates that cadets are comfortable with 
web-enabled curriculum support materials, and 
would be not be adverse to future 
developments in this area. 

Even though Courselnfo was housed on an 
intranet server inside the USAFA firewall, the 
largest problem cited by cadets was the 
reliability of the connection to the web-server. 
Cadets reported error messages that they 
misinterpreted to mean the Courselnfo server 
was down over the duration of the entire 
semester. However, the server was down only 
one time when classes were in session during the 
duration of the study (April to December, 2000). 
Error messages were more than likely caused by 
other connections within USAFAnet outside the 
research lab housing the Courselnfo server. 
Cadets, like most computer users, lack the time, 
knowledge and adequate system information to 
properly troubleshoot the problem, and 
therefore any network problem equally impedes 
their ability to access the information and use the 
system. Before any CMS can be deployed at the 
enterprise level, this problem will have to be 
examined further and even the perception of 
interrupted server access must be resolved. 

3.4.4 End of course critique evaluations for instructors 
using Blackboard Courselnfo 


end of course assessments before they 
volunteered to use Courselnfo, as well as after 
they began using the product. Their scores were 
higher for questions that might depend on 
curriculum delivery method, such as those 
pertaining to course organization and the ability 
to provide clear, well-organized instruction as 
well as questions that should be independent of 
using Courselnfo, such as the ability of the 
instructor to be a role model and knowledge of 
course material. This indicates that the software 
was evaluated by some of the "better" teaching 
faculty, and that using the software did not 
prevent them from providing quality instruction 
to the cadets. In addition, most of the faculty 
participants had higher assessment scores after 
they started using Courselnfo than before, but it is 
unclear whether that was due to the structure 
provided to the course from the software 
product, the web-enabling of the course itself, 
the expertise that comes with teaching a course 
multiple times, the increased attention and 
enthusiasm paid to the course by the instructors 
as they tried a new teaching method, or some 
other variable. 

A direct comparison of four instructors who used 
Courselnfo to provide web-enabled content to 
four instructors who taught sections of the same 
course, but did not provide any supplemental 
web-based materials, showed that the 
instructors using Courselnfo had higher end of 
semester course critique scores than the other 
group. The higher scores included clarity of 
course objectivesand quality and usefulness of 
course texts. This may be due to 
supplementation of the course text materials 
with online readings and examples. These 
instructors also received higher ratings for being 
a role model, which should not be dependent 
on using web-enabled curriculum. It is difficult to 
tease apart the various factors that contribute to 
higher ratings, and it may be that the course 
critiques for these sections have nothing to do 
with using the technology used to enhance the 
courses. However, it does appear from these 
results that this technology does not detract from 
an instructor's ability to provide quality instruction. 


The faculty members who chose to participate in 
this study scored higher than most instructors on 


l-manager’s Journal of Educational Technology, Vol. 2 • No. 1 • April-June 2005 99 






RESEARCH PAPERS 


Section 4: Overall Discussion and Conclusions 

Faculty members and cadets found that course 
management software (CMS) provided a satisfactory 
mechanism for web-enabled curriculum delivery to 
supplement traditional classroom instruction at USAFA. 
Faculty were able to learn one software package for web- 
based curriculum, assessments, collaborative features 
and for communicating with their students. They did not 
have to learn how to use HTML, and as illustrated by the 
course critique assessment data, the technology did not 
overshadow the teaching and learning of course content. 

If we had only focused on checklists of what the software 
products could do,then Blackboard Courselnfo and 
WebCT would have been considered equal, and Intralearn 
would have been a distant third. However, Blackboard 
Courselnfo received higher scores than the other products 
when ratings were based on user analysis of ease of use 
and usefulness. 

Many of the features found in these products such as 
discussion groups, student web pages and collaborative 
work areas that contributed to their high number of features 
were not widely used, nor deemed important by both 
faculty and cadets. Although some faculty may eventually 
use these features as they become more familiar with the 
software and pedagogy, at this time a CMS with an easy to 
use interface that contains a grade book, automated 
quizzes and a place to put announcements and course 
documents should be preferred to one that contains many 
collaboration features, yet has a difficult navigational 
interface or hard to use development tools. Once the 
faculty has incorporated the technology into the 
curriculum, there is a higher probability that the cadets 
would then find it useful. 

As predicted, the faculty and cadets had different 
requirements of the CMS product for developing and 
accessing resources, respectively. Therefore, if only the 
faculty's requirements are considered, or if the product is 
not evaluated by all of the user groups, the probability of 
purchasing a sub-optimal product for a majorily of the 
users increases. Although the requirements as stated by the 
faculty and cadets who had been using a CMS for an entire 
semester were more similar than those gathered from the 
naive users, often enterprise CMS purchase decisions are 
made priortothe purchasing of any system. 


To fully take advantage of the benefits of a CMS, there must 
be buy in from as much of the institution as possible, and 
course information should be integrated with other 
institutional databases into one system using an academic 
porfal or other similar interface. Since one advantage of 
this type of software is that the students only have to learn 
one navigational interface and information management 
system, those benefits may be decreased when more than 
one system with different navigational and features is 
deployed. However, to effectively deploy only one system 
here at USAFA, we will need to be able to effectively share 
content among instructors teaching the same course, and 
create grade books, announcements and other materials 
for specific sections of multi-section courses without 
duplication of the entire course. 

Section 5: Phase Two oftheCPEG Portal Project 

The results obtained in this study will help us determine the 
requirements for procuring course management system 
software for use here at USAFA. Giving faculty an efficient 
means of entering their homework assignments into a 
database system is the first and probably most important 
step to achieving the vision of an integrated portal system 
that includes the resources cadets need most; access to 
homework and reading assignments, course materials and the 
abililytomonitorlheirown progress through checking their grades. 

Following along the results of this research, the portal 
system that is being developed will incorporate a user- 
based design and analysis taking into account the needs 
of all users. To maximize efficiency, we will use a database 
model to facilitate the sharing and reusability of content. 
The system will be organized around a calendar that will 
tap into many scheduling databases currently housed 
independently here at USAFA as well as the course 
management system (Figure 3). To maximize ease of use, 
the system will have a single sign-on and use the inverse 
click rule to insure that data most important to the user will 
take the fewest number of mouse clicks to access. 

The portal design will break up the desktop in to a series of 
seamless blocks or portlets. This will allow the information 
displayed on the desktop to be personalized to the user 
and optional blocks such as personal links and task lists can 
be customized by the user. Although this project is still under 
development, we anticipate a prototype to be developed 
during Summer 2001 and undergo usability testing during 
the 2001 -02 academic year. 
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Figure 3: Prototype design for the USAFA portal or Cadet 
Personalized Educational Gateway system. 
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